Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: Simulation is a useful tool in situations where training data for machine learning models is costly to annotate or even hard to acquire. In this work, we propose a reinforcement learning-based method for automatically adjusting the parameters of any (non-differentiable) simulator, thereby controlling the distribution of synthesized data in order to maximize the accuracy of a model trained on that data. In contrast to prior art that hand-crafts these simulation parameters or adjusts only parts of the available parameters, our approach fully controls the simulator with the actual underlying goal of maximizing accuracy, rather than mimicking the real data distribution or randomly generating a large volume of data. We find that our approach (i) quickly converges to the optimal simulation parameters in controlled experiments and (ii) can indeed discover good sets of parameters for an image rendering simulator in actual computer vision applications. In order to train deep neural networks, significant effort has been directed towards collecting largescale datasets for tasks such as machine translation BID16, image recognition BID7 or semantic segmentation BID14 BID6. It is, thus, natural for recent works to explore simulation as a cheaper alternative to human annotation BID11 BID19 BID18. Besides, simulation is sometimes the most viable way to acquire data for rare events such as traffic accidents. However, while simulation makes data collection and annotation easier, it is still an open question what distribution should be used to synthesize data. Consequently, prior approaches have used human knowledge to shape the generating distribution of the simulator BID20, or synthesized large-scale data with random parameters BID18. In contrast, this paper proposes to automatically determine simulation parameters such that the performance of a model trained on synthesized data is maximized. Traditional approaches seek simulation parameters that try to model a distribution that resembles real data as closely as possible, or generate enough volume to be sufficiently representative. By learning the best set of simulation parameters to train a model, we depart from the above in three crucial ways. First, the need for laborious human expertise to create a diverse training dataset is eliminated. Second, learning to simulate may allow generating a smaller training dataset that achieves similar or better performances than random or human-synthesized datasets BID18, thereby saving training resources. Third, it allows questioning whether mimicking real data is indeed the best use of simulation, since a different distribution might be optimal for maximizing a test-time metric (for example, in the case of events with a heavy-tailed distribution).More formally, a typical machine learning setup aims to learn a function h θ that is parameterized by θ and maps from domain X to range Y, given training samples (x, y) ∼ p(x, y). Data x usually arises from a real world process (for instance, someone takes a picture with a camera) and labels y are often annotated by humans (someone describing the content of that picture). The distribution p(x, y) is assumed unknown and only an empirical sample D = {(x i, y i)} N i=1 is available. The simulator attempts to model a distribution q(x, y; ψ). In prior works, the aim is to adjust the form of q and parameters ψ to mimic p as closely as possible. In this work, we attempt to automatically learn the parameters of the simulator ψ such that the loss L of a machine learning model h θ is minimized over some validation data set D val. This objective can be formulated as the bi-level optimization problem DISPLAYFORM0 where h θ is parameterized by model parameters θ, D q(x,y| ψ) describes a data set generated by the simulator and θ(ψ) denotes the implicit dependence of the model parameters θ on the model's training data and consequently, for synthetic data, the simulation parameters ψ. In contrast to BID10, who propose a similar setup, we focus on the actual data generation process q(x, y; ψ) and are not limited to selecting subsets of existing data. In our formulation, the upper-level problem (equation 1a) can be seen as a meta-learner that learns how to generate data (by adjusting ψ) while the lower-level problem (equation 1b) is the main task model (MTM) that learns to solve the actual task at hand. In Section 2, we describe an approximate algorithm based on policy gradients BID24 to optimize the objective 1. For our algorithm to interact with a black-box simulator, we also present an interface between our model's output ψ and the simulator input. In various experiments on both toy data and real computer vision problems, Section 4 analyzes different variants of our approach and investigates interesting questions, such as: "Can we train a model h θ with less but targeted high-quality data?", or "Are simulation parameters that approximate real data the optimal choice for training models?". The experiments indicate that our approach is able to quickly identify good scene parameters ψ that compete and in some cases even outperform the actual validation set parameters for synthetic as well as real data, on computer vision problems such as object counting or semantic segmentation. Given a simulator that samples data as (x, y) ∼ q(x, y; ψ), our goal is to adjust ψ such that the MTM h θ trained on that simulated data minimizes the risk on real data (x, y) ∼ p(x, y). Assume we are given a validation set from real data D val and we can sample synthetic datasets D q(x,y| ψ) ∼ q(x, y| ψ). Then, we can can train h θ on D q(x,y| ψ) by minimizing equation 1b. Note the explicit dependence of the trained model parameters θ * on the underlying data generating parameters ψ in equation 1b. To find ψ *, we minimize the empirical risk over the held-out validation set D val, as defined in equation 1a. Our desired overall objective function can thus be formulated as the bi-level optimization problem in equation 1.Attempting to solve it with a gradient-based approach poses multiple constraints on the lower-level problem 1b like smoothness, twice differentiability and an invertible Hessian BID2 BID5. For our case, even if we choose the model h θ to fulfill these constraints, the objective would still be non-differentiable as we (i) sample from a distribution that is parameterized by the optimization variable and (ii) the underlying data generation process (e.g., an image rendering engine) is assumed non-differentiable for the sake of generality of our approach. In order to cope with the above defined objective, we resort to policy gradients BID24 to optimize ψ. Our goal is to generate a synthetic dataset such that the main task model (MTM) h θ, when trained on this dataset until convergence, achieves maximum accuracy on the test set. The test set is evidently not available during train time. Thus, the task of our algorithm is to maximize MTM's performance on the validation set by generating suitable data. Similar to reinforcement learning, we define a policy π ω parameterized by ω that can sample parameters ψ ∼ π ω for the simulator. The simulator can be seen as a generative model G(x, y| ψ) which generates a set of data samples (x, y) conditioned on ψ. We provide more details on the interface between the policy and the data generating function in the following section and give a concrete example for computer vision applications in Section 4. The Reward R Figure 1: A high-level overview of our "learning to simulate" approach. A policy π ω outputs parameters ψ which are used by a simulator to generate a training dataset. The main task model (MTM) is then trained on this dataset and evaluated on a validation set. The obtained accuracy serves as reward signal R for the policy on how good the synthesized dataset was. The policy thus learns how to generate data to maximize the validation accuracy.policy receives a reward that we define based on the accuracy of the trained MTM on the validation set. Figure 1 provides a high-level overview. Specifically, we want to maximize the objective DISPLAYFORM0 with respect to ω. The reward R is computed as the negative loss L or some other accuracy metric on the validation set. Following the REINFORCE rule we obtain gradients for updating ω as DISPLAYFORM1 An unbiased, empirical estimate of the above quantity is DISPLAYFORM2 whereÂ k = R(ψ k) − b is the advantage estimate and b is a baseline that we choose to be an exponential moving average over previous rewards. In this empirical estimate, K is the number of different datasets D q(x,y| ψ k) sampled in one policy optimizing batch and R(ψ k) designates the reward obtained by the k-th MTM trained until convergence. Given the basic update rule for the policy π ω, we can design different variants of our algorithm for learning to simulate data by introducing three control knobs. First, we define the number of training epochs ξ of the MTM in each policy iteration as a variable. The intuition is that a reasonable reward signal may be obtained even if MTM is not trained until full convergence, thus reducing computation time significantly. Second, we define the size M of the data set generated in each policy iteration. Third, we either choose to retain the MTM parameters θ from the previous iteration and fine-tune on the newly created data or we estimate θ from scratch (with a random initialization). This obviously is a trade-off because by retaining parameters the model has seen more training data in total but, at the same time, may be influenced by suboptimal data in early iterations. We explore the impact of these three knobs in our experiments and appendix. Algorithm 1 summarizes our approach. DISPLAYFORM3 Train or fine-tune K main task models (MTM) for ξ epochs on data provided by M k Obtain rewards R(ψ k), i.e., the accuracy of the trained MTMs on the validation set Compute the advantage estimateÂ k = R(ψ k) − b Update the policy parameters ω via equation 4 end Algorithm 1: Our approach for "learning to simulate" based on policy gradients. We defined a general black-box simulator as a distribution G(x, y| ψ) over data samples (x, y) parameterized by ψ. In practice, a simulator is typically composed of a deterministic "rendering" process R and a sampling step as DISPLAYFORM0, where the actual data description ρ (e.g., what objects are rendered in an image) is sampled from a distribution S parametrized by the provided simulation parameters ψ and specific rendering settings φ (e.g., lighting conditions) are sampled from a distribution P also parameterized by ψ. To enable efficient sampling (via ancestral sampling) BID1, the data description distribution is often modeled as a Bayesian network (directed acyclic graph) where ψ defines the parameters of the distributions in each node, but more complex models are possible too. The interface to the simulator is thus ψ which describes parameters of the internal probability distributions of the black-box simulator. Note that ψ can be modeled as an unconstrained continuous vector and still describe various probability distributions. For instance, a continuous Gaussian is modeled by its mean and variance. A K-dimensional discrete distribution is modeled with K real values. We assume the black-box normalizes the values to a proper distribution via a softmax. With this convention all input parameters to the simulator are unconstrained continuous variables. We thus model our policy as the multivariate Gaussian π ω (ρ, φ) = N (ω, σ 2) with as many dimensions as the sum of the dimensions of parameters ρ and φ. For simplicity, we only optimize for the mean and set the variance to 0.05 in all cases, although the policy gradients defined above can handle both. Note that our policy can be extended to a more complicated form, e.g., by including the variance. The proposed approach can be seen as a meta-learner that alters the data a machine learning model is trained on to achieve high accuracy on a validation set. This concept is similar to recent papers that learn policies for neural network architectures BID25 and optimizers BID0. In contrast to these works, we are focusing on the data generation parameters and actually create new, randomly sampled data in each iteration. While BID10 proposes a subset selection approach for altering the training data, we are actually creating new data. This difference is important because we are not limited to a fixed probability distribution at data acquisition time. We can thus generate or oversample unusual situations that would otherwise not be part of the training data. Similar to the above-mentioned papers, we also choose a variant of stochastic gradients (policy gradients BID24) to overcome the non-differentiable sampling and rendering and estimate the parameters of the policy π ω. While alternatives for black-box optimization exist, like evolutionary algorithms BID21 or sampling-based methods BID1, we favor policy gradients in this work for their sample efficiency and success in prior art. BID13 train a policy to generate a program that creates a copy of an input image. Similar to us, they use policy gradients to train the policy, although they use an adversarial loss to construct their reward. Again, BID15 seek to tune parameters of a simulator such that the marginal distribution of the synthetic data matches the distribution of the observed data. In contrast to both works, we learn parameters of a simulator that maximize performance of a main task model on a specific task. The learned distribution need not match the distribution of the observed data. When relying on simulated data for training machine learning models, the issue of "domain gap" between real and synthetic data arises. Many recent works BID12 BID23 focus on bridging this domain gap, particularly for computer vision tasks. Even if we are able to tune parameters perfectly, there exists a simulation-to-real domain gap which needs to be addressed. Thus, we believe the contributions of our work are orthogonal. The intent of our experimental evaluation is (i) to illustrate the concept of our approach in a controlled toy experiment (section 4.1), (ii) to analyze different properties of the proposed algorithm 1 on a high-level computer vision task (section 4.3) and (iii) to demonstrate our ideas on real data for semantic image segmentation (section 4.5). The decision boundaries (shaded areas) of a non-linear SVM trained on data generated by q(x, y| ψ i) for three different iterations i of our policy π ω. The data points overlaid are the test set. Bottom row: Decision boundary when trained on data sampled from p(x, y| ψ real) (left) and on the converged parameters ψ * (middle); Data sampled from q(x, y| ψ *) (right). To illustrate the concept of our proposed ideas we define a binary classification task on the 2-dimensional Euclidean space, where data distribution p(x, y| ψ real) of the two classes is represented by Gaussian mixture models (GMM) with 3 components, respectively. We generate validation and test sets from p(x, y| ψ real). Another GMM distribution q(x, y| ψ) reflects the simulator that generates training data for the main task model (MTM) h θ, which is a non-linear SVM with RBF-kernels in this case. To demonstrate the practical scenario where a simulator is only an approximation to the real data, we fix the number of components per GMM in ψ to be only 2 and let the policy π ω only adjust mean and variances of the GMMs. Again, the policy adjusts ψ such that the accuracy (i.e., reward R) of the SVM is maximized on the validation set. The top row of figure 2 illustrates how the policy gradually adjusts the data generating distribution q(x, y| ψ) such that reward R is increased. The learned decision boundaries in the last iteration (right) well separate the test data. The bottom row of figure 2 shows the SVM decision boundary when trained with data sampled from p(x, y| ψ real) (left) and with the converged parameters ψ * from the policy (middle). The third figure in the bottom row of figure 2 shows samples from q(x, y| ψ *). The sampled data from the simulator is clearly different than the test data, which is obvious given that the simulator's GMM has less components per class. However, it is important to note that the decision boundaries are still learned well for the task at hand. For the following experiments we use computer vision applications and thus require a generative scene model and an image rendering engine. We focus on traffic scenes as simulators/games for this scenario are publicly available (CARLA BID8 with Unreal engine as backend). However, we need to note that substantial extensions were required to actually generate different scenes according to a scene model rather than just different viewpoints of a static map. Many alternative simulators like BID17 BID18 BID22 are similar where an agent can navigate a few pre-defined maps, but the scene itself is not parameterized and cannot be changed on the fly. To actually synthesize novel scenes, we first need a model S(ρ | ψ) that allows to sample instances of scenes ρ given parameters ψ of the probability distributions of the scene model. Recall that ψ is produced by our learned policy π ω. Our traffic scene model S(ρ | ψ) handles different types of intersections, various car models driving on the road, road layouts and buildings on the side. Additionally our rendering model P (φ | ψ) handles shows the reward on the unseen test set. We observe that even when using the "adversarial" initialization of parameters, our approach converges to the same reward R, but at a slower rate. (c) shows the reward on the unseen test for different random parameter initializations.weather conditions. Please see the appendix for more details. In our experiments, the model is free to adjust some of the variables, e.g., the probability of cars being on the road, weather conditions, etc. Given these two distributions, we can sample a new scene and render it as R(S(ρ | ψ), P (φ | ψ)). Figure 3 shows examples of rendered scenes. As a first high-level computer vision task we choose counting cars in rendered images, where the goal is to train a convolutional neural network h θ to count all instances individually for five types of cars in an image. The evaluation metric and (also the loss) is the 1 distance between predicted and ground truth count, averaged over the different car types. The reward R is the negative 1 loss. For this experiment, we generate validation and test sets with a fixed and pre-defined distribution ψ real.Initialization: We first evaluate our proposed policy (dubbed "LTS" in the figures) for two different initializations, a "standard" random one and an initialization that is deliberately picked to be suboptimal (dubbed "adversarial"). We also compare with a model trained on a data set sampled with ψ real, i.e., the test set parameters. Figure 4 explains our . We can see in figure 4a that our policy π ω ("LTS") quickly reaches high reward R on the validation set, equal to the reward we get when training models with ψ real. Figure 4b shows that high rewards are also obtained on the unseen test set for both initializations, albeit convergence is slower for the adversarial initialization. The reward after convergence is comparable to the model trained on ψ real. Since our policy π ω is inherently stochastic, we show in figure 4c convergence for different random initializations and observe a very stable behavior. Accumulating data: Next, we explore the difference between training the MTM h θ from scratch in each policy iteration or retaining its parameters and fine-tune, see algorithm 1. We call the second option the "accumulated main task model (AMTM)" because it is not re-initialized and accumulates information over policy iterations. The intention of this experiment is to analyze the situation where the simulator is used for generating large quantities of data, like in BID18. First, by comparing figures 4b and 5a, we observe that the reward R gets significantly higher than when training MTM from scratch in each policy iteration. Note that we still use the MTM reward as our training signal, we only observe the AMTM reward for evaluation purposes. For the case of accumulating the MTM parameters, we further compare with two baselines. First, replicating a hand-crafted choice of simulation parameters, we assume no domain expertise and (a) (b) (c) Figure 5: (a) Reward R of the accumulated main task model on the car-counting task using different training schemes. We observe that training an accumulated main task network using a learning policy at each step is superior to training it with a dataset generated either using the final parameters of the policy or random parameters. (b) Learning-to-simulate converges even with an "adversarial" initialization of parameters, albeit in more epochs. (c) Reward of the accumulated main task model using different number of training epochs ξ for h θ.randomly sample simulator parameters (within a sensible range) in each iteration ("random policy params"). Second, we take the parameters given by our learned policy after convergence ("final policy params"). For reference, we train another AMTM with the ground truth validation set parameters ("validation params") as our upper-bound. All baselines are accumulated main task models, but with fixed parameters for sampling data, i.e., resembling the case of generating large datasets. We can see from figure 5a that our approach gets very close to the ground truth validation set parameters and significantly outperforms the random parameter baseline. Interestingly, "LTS" even outperforms the "final policy params" baseline, which we attribute to increased variety of the data distribution. Again, "LTS" converges to a high reward R even with an adversarial initialization, see figure 5bNumber of epochs: Similar to the previous experiment, we now analyze the impact of the number of epochs ξ used to train the main task model h θ in the inner loop of learning to simulate. Figure 5c shows the reward of the accumulated MTM for four different values of ξ. Our , for the car-counting task, is that learning to simulate is robust to lower training epochs, which means that even if the MTM has not fully converged yet the reward signal is good enough to provide guidance for our system leading to a potential wall-time speed up of the overall algorithm. All four cases converge, including the one where we train the MTM for only one epoch. Note that this is dependent on the task at hand, and a more complicated task might necessitate convergence of the main task model to provide discriminative rewards. For the next set of experiments we use semantic segmentation as our test bed, which aims at predicting a semantic category for each pixel in a given RGB image BID3. Our modified CARLA simulator BID8 provides ground truth semantic segmentation maps for generated traffic scenes, including categories like road, sidewalk or cars. For the sake of these experiments, we focus on the segmentation accuracy of cars, measured as intersection-over-union (IoU), and allow our policy π ω to adjust scene and rendering parameters to maximize reward R (i.e., car IoU). This includes the probability of generating different types of cars, length of road and weather type. The main task model h θ is a CNN that takes a rendered RGB image as input and predicts a per-pixel classification output. We first generate validation set parameters ψ val that reflect traffic scenes moderately crowded with cars, unbalanced car types, random intersections and buildings on the side. As a reference point for our proposed algorithm, we sample a few data sets with the validation set parameters ψ val, train MTMs and report the maximum reward (IoU of cars) achieved. We compare this with our learned policy π ω and can observe in figure 6a that it actually outperforms the validation set parameters. This is an interesting observation because it shows that the validation set parameters ψ val may not always be the optimal choice for training a segmentation model. We demonstrate the practical impact of our learning-to-simulate approach on semantic segmentation on by training a main task model (MTM) h θ with a reward signal coming from real data. Using simulated data for semantic segmentation was recently investigated from a (a) (b) Figure 6: (a) Reward curves of our approach compared to a model trained on data generated with the actual validation set parameters on the synthetic semantic segmentation task. (b) Reward curves on the real validation set of KITTI for semantic segmentation. We plot the learning-to-simulate, the maximum reward achieved using random search and the maximum and mean using random parameters. All methods use 600 iterations. Training data random params random search LTS KITTI train set Car IoU 0.480 0.407 0.579 0.778 Table 1: Segmentation Car IoU on the unseen KITTI test set for a ResNet-50 segmentation network trained using synthetic data generated by random parameters or learned parameters using random search or learning to simulate (LTS) for 600 epochs of each. We test the epoch with highest validation reward on the KITTI test set. We also report the maximum car IoU obtained by training on 982 annotated real KITTI training images.domain adaptation perspective BID23 BID18, where an abundant set of simulated data is leveraged to train models applicable on real data. Here, we investigate targeted generation of simulated data and its impact on real data. Since the semantic label space of KITTI and our CARLA simulator are not identical, we again focus on segmentation of cars by measuring IoU for that category. For our main task model h θ we use a CNN that takes a rendered RGB image as input and predicts a per-pixel classification output with a ResNet-50 backbone. As our baseline we train the main task model separately 600 times, with data generated by the simulator using different sets of random parameters for each one. We monitor the validation Car IoU metric for each of these networks and pick the one with highest validation reward. We then test it on the unseen KITTI test set and report the Car IoU in table 1. For illustration purposes we show the reward curve of our approach on the validation set as well as the maximum for random search and the maximum and mean for random parameters in Figure 6b.However, it is important to mention that parameters which are actually good for training an MTM h θ are unknown, making our automatic approach attractive in such situations. The on the unseen real KITTI test set in table 1 confirm the superior of learning-to-simulate. We train using synthetic data generated by random or learned parameters for 600 epochs of each. We pick the epoch with highest validation reward and test it on the KITTI test set. For reference, we also report the maximum car IoU obtained by our network by training on 982 annotated real KITTI training images. Additionally, we verify empirically that parameter optimization using policy gradients (learning to simulate) outperforms random search for this problem. Results are reported in table 1. Learning to simulate can be seen as a meta-learning algorithm that adjusts parameters of a simulator to generate synthetic data such that a machine learning model trained on this data achieves high accuracies on validation and test sets, respectively. Given the need for large-scale data sets to feed deep learning models and the often high cost of annotation and acquisition, we believe our approach is a sensible avenue for practical applications to leverage synthetic data. Our experiments illustrate the concept and demonstrate the capability of learning to simulate on both synthetic and real data. For future work, we plan to expand the label space in our segmentation experiments, apply the algorithm to other tasks like object detection and to explore a dynamic memory of previously generated data for improving our learning to simulate procedure. A TRAFFIC SCENE MODEL Our model comprises the following elements:• A straight road of variable length.• Either an L, T or X intersection at the end of the road.• Cars of 5 different types which are spawned randomly on the straight road.• Houses of a unique type which are spawned randomly on the sides of the road.• Four different types of weather. All of these elements are tied to parameters: ρ k can be decomposed into parameters which regulate each of these objects. The scene is generated "block" by "block". A block consists of a unitary length of road with sidewalks. Buildings can be generated on both sides of the road and cars can be generated on the road. ρ k,car designates the probability of car presence in any road block. Cars are sampled block by block from a Bernouilli distribution X ∼ Bern (ρ k,car). To determine which type of car is spawned (from our selection of 5 cars) we sample from a Categorical distribution which is determined by 5 parameters ρ k,cari where i is an integer representing the identity of the car and i ∈. ρ k,house designates the probability of house presence in any road block. Houses are sampled block by block from a Bernouilli distribution X ∼ Bern (ρ k,house).Length to intersection is sampled from a Categorical distribution determined by 10 parameters ρ k,lengthi with i ∈ where i denotes the length from the camera to the intersection in "block" units. Weather is sampled randomly from a Categorical distribution determined by 4 parameters φ k,weatheri where i is an integer representing the identity of the weather and i ∈. L, T and X intersections are sampled randomly with equal probability. In table 2, we present classification accuracy for the toy problem in Section 4.1 with Q distributions using different number of gaussians. We can observe that by using learning to simulate we obtain better classification than using a dataset generated using the test set parameters (mean and variance of gaussians in P distribution). In this section we visualize learning of parameters in the car counting problem described in Section 4.3. In particular we show how the parameters of weather type and car type evolve in time in FIG3. We explore the parameter M of our algorithm that controls the dataset size generated in each policy iteration. For example, when M = 100, we generate at each policy step a dataset of 100 images using the parameters from the policy which are then used to train our main task model h θ. We evaluate policies with sizes 20, 50, 100 and 200. In figure 8 we show a comparative graph of final errors on the validation and test sets for different values of M. For a fair comparison, we generate 40,000 images with our final learned set of parameters and train h θ for 5 epochs and evaluate on the test set. We observed that for this task a dataset size of just 20 suffices for our model to converge to good scene Figure 8: Test and validation error from main task networks trained on 40,000 images during 5 epochs using final learned parameters for different sizes of datasets generated per policy iteration.parameters ψ, which is highly beneficial for the wall-time convergence speed. Having less data per policy iteration means faster training of the MTM h θ. Since our method is stochastic in nature we verify that "learning to simulate" converges in the car counting task using different random seeds. We observer in figure 9a that the reward converges to the same value with three different random seeds. Additionally, in figure 9b, we observe that the accumulated main task network test reward also converges with different random seeds.(a) (b) Figure 9: Main task network reward and accumulated MTN reward converge using different random seeds.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose an algorithm that automatically adjusts parameters of a simulation engine to generate training data for a neural network such that validation accuracy is maximized.

Represent the short summary for finding the corresponding scientific article (long)

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the scientific article to find a short TLDR

Question: Modelling statistical relationships beyond the conditional mean is crucial in many settings. Conditional density estimation (CDE) aims to learn the full conditional probability density from data. Though highly expressive, neural network based CDE models can suffer from severe over-fitting when trained with the maximum likelihood objective. Due to the inherent structure of such models, classical regularization approaches in the parameter space are rendered ineffective. To address this issue, we develop a model-agnostic noise regularization method for CDE that adds random perturbations to the data during training. We demonstrate that the proposed approach corresponds to a smoothness regularization and prove its asymptotic consistency. In our experiments, noise regularization significantly and consistently outperforms other regularization methods across seven data sets and three CDE models. The effectiveness of noise regularization makes neural network based CDE the preferable method over previous non- and semi-parametric approaches, even when training data is scarce. While regression analysis aims to describe the conditional mean E[y|x] of a response y given inputs x, many problems such as risk management and planning under uncertainty require gaining insight about deviations from the mean and their associated likelihood. The stochastic dependency of y on x can be captured by modeling the conditional probability density p(y|x). Inferring such a density function from a set of empirical observations {(x n, y n)} N n=1 is typically referred to as conditional density estimation (CDE) and is the focus of this paper. In the recent machine learning literature, there has been a resurgence of interest in high-capacity density models based on neural networks (; ;). Since this line of work mainly focuses on the modelling of images based on large scale data sets, over-fitting and noisy observations are of minor concern in this context. In contrast, we are interested in CDE in settings where data may be scarce and noisy. When combined with maximum likelihood estimation, the flexibility of such high-capacity models in over-fitting and poor generalization. While regression typically assumes Gaussian conditional noise, CDE uses expressive distribution families to model deviations from the conditional mean. Hence, the overfitting problem tends to be even more severe in CDE than in regression. Classical regularization of the neural network weights such as weight decay has been shown to be effective for regression and classification. However, in the context of CDE, the output of the neural network merely controls the parameters of a density model such as a Gaussian Mixture or Normalizing Flow. This makes the standard regularization methods in the parameter space less effective and harder to analyze. Aiming to address this issue, we propose and analyze noise regularization, a method well-studied in the context of regression and classification, for the purpose of conditional density estimation. In that, the paper attempts to close a gap in previous research. By adding small random perturbations to the data during training, the conditional density estimate is smoothed and tends to generalize better. In fact, we show that adding noise during maximum likelihood estimation is equivalent to penalizing the second derivatives of the conditional log-probability. Visually, the respective regularization term punishes very curved or even spiky density estimators in favor of smoother variants, which proves to be a favorable inductive bias in many applications. Moreover, under some regularity conditions, we show that the proposed regularization scheme is asymptotically consistent, converging to the unbiased maximum likelihood estimator. This does not only support the soundness of the proposed method but also endows us with useful insight in how to set the regularization intensity relative to the data dimensionality and training set size. Overall, the proposed noise regularization scheme is easy to implement and agnostic to the parameterization of the CDE model. We empirically demonstrate its effectiveness on three different neural network based models. The experimental show that noise regularization outperforms other regularization methods significantly and consistently across various data sets. Finally, we demonstrate that, when properly regularized, neural network based CDE is able to improve upon state-of-the art non-parametric estimators, even when only 400 training observations are available. Density Estimation. Let X be a random variable with probability density function (PDF) p(x) defined over the domain X ⊆ R dx. Given a collection D = {x 1, ..., x n} of observations sampled from p(x), the goal is to find a good estimatef (x) of the true density function p. In parametric estimation, the PDFf is assumed to belong to a parametric family F = {f θ (·)|θ ∈ Θ} where the density function is described by a finite dimensional parameter θ ∈ Θ. The standard method for estimating θ is maximum likelihood estimation, wherein θ is chosen so that the likelihood of the data D is maximized. This is equivalent to minimizing the Kullback-Leibler divergence between the empirical data distribution p D (x) = 1 n n i=1 δ(||x − x i ||) (i.e., mixture of point masses in the observations x i) and the parametric distributionf θ: From a geometric perspective, can be viewed as an orthogonal projection of p D (x) onto F w.r.t. the reverse KL-divergence. Hence, is also commonly referred to as an M-projection . In contrast, non-parametric density estimators make implicit smoothness assumptions through a kernel function. The most popular non-parametric method, kernel density estimation (KDE), places a symmetric density function K(z), the so-called kernel, on each training data point x n . The ing density estimate reads asq(. Beyond the appropriate choice of K(·), a central challenge is the selection of the bandwidth parameter h which controls the smoothness of the estimated PDF . Conditional Density Estimation (CDE). Let (X, Y) be a pair of random variables with respective domains X ⊆ R dx and Y ⊆ R dy and realizations x and y. Let p(y|x) = p(x, y)/p(x) denote the conditional probability density of y given x. Typically, Y is referred to as a dependent variable (explained variable) and X as conditional (explanatory) variable. Given a dataset of observations D = {(x n, y n)} N n=1 drawn from the joint distribution (x n, y n) ∼ p(x, y), the aim of conditional density estimation (CDE) is to find an estimatef (y|x) of the true conditional density p(y|x). In the context of CDE, the KL-divergence objective is expressed as expectation over p(x): Corresponding to, we refer to the minimization of w.r.t. θ as conditional M-projection. Given a dataset D drawn i.i.d. from p(x, y), the conditional MLE following from can be stated as 3 RELATED WORK The first part of this section discusses relevant work in the field of CDE, focusing on high-capacity models that make little prior assumptions. The second part relates our approach to previous regularization and data augmentation methods. Non-parametric CDE. A vast body of literature in statistics and econometrics studies nonparametric kernel density estimators (KDE) and the associated bandwidth selection problem, which concerns choosing the appropriate amount of smoothing (; ;). To estimate conditional probabilities, previous work proposes to estimate both the joint and marginal probability separately with KDE and then computing the conditional probability as their ratio . Other approaches combine non-parametric elements with parametric elements (; ;). Despite their theoretical appeal, non-parametric density estimators suffer from poor generalization in regions where data is sparse (e.g., tail regions), causing rapid performance deterioration as the data dimensionality increases . CDE based on neural networks. Most work in machine learning focuses on flexible parametric function approximators for CDE. In our experiments, we use the work of and , who propose to use a neural network to control the parameters of a mixture density model. A recent trend in machine learning are latent density models such as cGANs and cVAEs . Although such methods have been shown successful for estimating distributions of images, the probability density function (PDF) of such models is intractable. More promising in this sense are normalizing flows (; ;), since they provide the PDF in tractable form. We employ a neural network controlling the parameters of a normalizing flow as our third CDE model to showcase the empirical efficacy of our regularization approach. Regularization. Since neural network based CDE models suffer from severe over-fitting when trained with the MLE objective, they require proper regularization. Classical regularization of the parameters such as weight decay (; ;), l 1 /l 2 -penalties and Bayesian priors have been shown to work well in the regression and classification setting. However, in the context of CDE, it is less clear what kind of inductive bias such a regularization imposes on the density estimate. In contrast, our regularization approach is agnostic w.r.t. parametrization and is shown to penalize strong variations of the log-density function. Regularization methods such as dropout are closely related to ensemble methods . Thus, they are orthogonal to our work and can be freely combined with noise regularization. Adding noise during training. Adding noise during training is a common scheme that has been proposed in various forms. This includes noise on the neural network weights or activations (; ;) and additive noise on the gradients for scalable MCMC posterior inference . While this line of work corresponds to noise in the parameter space, other research suggests to augment the training data through random and/or adversarial transformations of the data (; Burges & Schölkopf, 1996; ;). Our approach transforms the training observations by adding small random perturbations. While this form of regularization has been studied in the context of regression and classification problems (a; ; ; ;), this paper focuses on the regularization of CDE. In particular, we build on top of the of showing that training with noise corresponds to a penalty on strong variations of the log-density and extend previous consistency for regression of Holmstrom & Koistinen (1992a) to the more general setting of CDE. To our best knowledge, this is also the first paper to evaluate the empirical efficacy of noise regularization for density estimation. When considering expressive families of conditional densities, standard maximum likelihood estimation of the model parameters θ is ill suited. As can be observed in Figure 1, simply minimizing the negative log-likelihood of the data leads to severe over-fitting and poor generalization beyond the training data. Hence, it is necessary to impose additional inductive bias, for instance, in the form of regularization. Unlike in regression or classification, the form of inductive bias imposed by popular regularization techniques such as weight decay (; Kukačka et al., 2017) is less clear in the CDE setting, where the neural network weights often only indirectly control the probability density through a unconditional density model, e.g., a Gaussian Mixture. We propose to add noise perturbations to the data points during the optimization of the log-likelihood objective. This can be understood as replacing the original data points (x i, y i) by random variables tions K x (ξ x) and K y (ξ y) respectively. Further, we choose the noise to be zero centered as well as identically and independently distributed among the data dimensions, with standard deviation h: This can be seen as data augmentation, where "synthetic" data is generated by randomly perturbing the original data. Since the supply of noise vectors is technically unlimited, an arbitrary large augmented data set can be generated by repetitively sampling data points from D, and adding a random perturbation vector to the respective data point. This procedure is formalized in Algorithm 1. For notational brevity, we set Z:= X × Y, z:= (x, y) and denotef θ (z):=f θ (y|x). The presented noise regularization approach is agnostic to whether we are concerned with unconditional or conditional MLE. Thus, the generic notation also allows us to generalize the to both settings (derived in the remainder of the paper). Require: D = {z 1, ..., z n}, noise intensity h Require: number of perturbed samples r, 1: for j = 1 to r do 2: Select i ∈ {1, ..., n} with equal prob. Draw perturbation ξ ∼ K 6: When considering highly flexible parametric families such as Mixture Density Networks (MDNs) , the maximum likelihood solution in line 5 of Algorithm 1 is no longer tractable. In such case, one typically resorts to numerical optimization techniques such as mini-batch gradient descent and variations thereof. In this context, the generic procedure in Algorithm 1 can be transformed into a simple extensions of mini-batch gradient descent on the MLE objective (see Algorithm 2). Specifically, each mini-batch is perturbed with i.i.d. noise before computing the MLE objective function (forward pass) and the respective gradients (backward pass). Intuitively, the previously presented variable noise can be interpreted as "smearing" the data points during the maximum likelihood estimation. This alleviates the jaggedness of the density estimate arising from an un-regularized maximum likelihood objective in flexible density classes. We will now give this intuition a formal foundation, by mathematically analyzing the effect of the noise perturbations. Before discussing the particular effects of randomly perturbing the data during conditional maximum likelihood estimation, we first analyze noise regularization in a more general case. Let l (D) be a loss function over a set of data points D = {z 1, ..., z n}, which can be partitioned into a sum of losses l(D) = n i=1 l(z i), corresponding to each data point z i: The expected loss l(z i +ξ), ing from adding random perturbations, can be approximated by a second order Taylor expansion around z i. Using the assumption about ξ in, the expected loss an be written as where l(z i) is the loss without noise and ∂z 2 (z) zi the Hessian of l w.r.t z, evaluated at z i. Assuming that the noise ξ is small in its magnitude, O(ξ 3) is negligible. This effect has been observed earlier by and. See Appendix A for derivations. When concerned with maximum likelihood estimation of a conditional densityf θ (y|x), the loss function coincides with the negative conditional log-likelihood l(y i, x i) = − logf θ (y i |x i). Let the standard deviation of the additive data noise ξ x, ξ y be h x and h y respectively. Maximum likelihood estimation (MLE) with data noise is equivalent to minimizing the loss In that, the first term corresponds to the standard MLE objective, while the other two terms constitute a form of smoothness regularization. The second term in penalizes large negative second derivatives of the conditional log density estimate logf θ (y|x) w.r.t. y. As the MLE objective pushes the density estimate towards high densities and strong concavity in the data points y i, the regularization term counteracts this tendency to over-fit and overall smoothes the fitted distribution. The third term penalizes large negative second derivatives w.r.t. the conditional variable x, thereby regularizing the sensitivity of the density estimate to changes in the conditional variable. The intensity of the noise regularization can be controlled through the variance (h 2 x and h 2 y) of the random perturbations. Figure 1 illustrates the effect of the introduced noise regularization scheme on MDN estimates. Plain maximum likelihood estimation (left) leads to strong over-fitting, ing in a spiky distribution that generalizes poorly beyond the training data. In contrast, training with noise regularization (center and right) in smoother density estimates that are closer to the true conditional density. We now establish asymptotic consistency for the proposed noise regularization. In particular, we show that, under some regularity conditions, concerning integrability and decay of the noise regularization, the solution of Algorithm 1 converges to the asymptotic MLE solution. a continuous function of z and θ. Moreover, we assume that the parameter space Θ is compact. In the classical MLE setting, the idealized loss, corresponding to a (conditional) M-projection of the true data distribution onto the parametric family, reads as As we typically just have a finite number of samples from p(z), the respective empirical estimatê is used as training objective. Note that we now define the loss as function of θ, and, for fixed θ, treat l n (θ) as a random variable. Under some regularity conditions, one can invoke the uniform law of large numbers to show consistency of the empirical ML objective in the sense that sup θ∈Θ |l n (θ) − l(θ)| a.s. − − → 0 (see Appendix B for details). In case of the presented noise regularization scheme, the maximum likelihood estimation is performed using on the augmented data {z j} rather than the original data {z i}. For our analysis, we view Algorithm 1 from a slightly different angle. In fact, the data augmentation procedure of uniformly selecting a data point from {z 1, ..., z n} and perturbing it with a noise vector drawn from K can be viewed as drawing i.i.d. samples from a kernel density estimateq. Hence, maximum likelihood estimation with variable noise can be understood as 1. forming a kernel density estimateq (h) n of the training data 2. followed by a (conditional) M-projection ofq (h) n onto the parametric family. In that, step 2 aims to find the θ * that minimizes the following objective: Since is generally intractable, r samples are drawn from the kernel density estimate, forming the following Monte Carlo approximation of which corresponds to the loss in line 5 Algorithm 1: We are concerned with the consistency of the training procedure in Algorithm 1, similar to the classical MLE consistency discussed above. Hence, we need to show that − − → 0 as n, r → ∞. We begin our argument by decomposing the problem into easier sub-problems. In particular, the triangle inequality is used to obtain the following upper bound: Note thatl n,r (θ) is based on samples from the kernel density estimate, which are obtained by adding random noise vectors ξ ∼ K(·) to our original training data. Since we can sample an unlimited amount of such random noise vectors, r can be chosen arbitrarily high. This allows us to make sup θ∈Θ |l n (θ)| arbitrary small by the uniform law of large numbers. In order to make sup θ∈Θ |l (h) n (θ) − l(θ)| small in the limit n → ∞, the sequence of bandwidth parameters h n needs to be chosen appropriately. Such can then be combined using a union bound argument. In the following we outline the steps leading us to the desired . In that, the proof methodology is similar to Holmstrom & Koistinen (1992b). While they show consistency for regression with a quadratic loss function, our proof deals with generic and inherently unbounded log-likelihood objectives and thus holds for a much more general class of learning problems. The full proofs can be found in the Appendix. Initially, we have to make asymptotic integrability assumptions that ensure that the expectations in l (h) n (θ) and l(θ) are well-behaved in the limit (see Appendix C for details). Given respective integrability, we are able to obtain the following proposition. Proposition 1 Suppose the regularity conditions and are satisfied, and that almost surely. In we find conditions on the asymptotic behavior of the smoothing sequence (h n). These conditions also give us valuable guidance on how to properly choose the noise intensity in line 4 of Algorithm 1 (see Section 4.3 for discussion). The in demonstrates that, under the discussed conditions, replacing the empirical data distribution with a kernel density estimate still in an asymptotically consistent maximum likelihood objective. However, as previously discussed, l n (θ) is intractable and, thus, replaced by its sample estimatel n,r. Since we can draw an arbitrary amount of samples fromq (h) n, we can approximate l (h) n (θ) with arbitrary precision. Given a fixed data set D of size n > n 0, this means that lim r→∞ sup θ∈Θ n (θ) = 0 almost surely, by and the uniform law of large numbers. Since our original goal was to also show consistency for n → ∞, this is combined with Proposition 1, obtaining the following consistency theorem. Theorem 1 Suppose the regularity conditions and are satisfied, h n fulfills and Θ is compact. Then, lim almost surely. In that, lim used to denote the limit superior ("lim sup") of a sequence. Training a (conditional) density model with noise regularization means minimizingl n,r (θ) w.r.t. θ. As of this optimization, one obtains a parameter vectorθ (h) n,r, which we hope is close to the minimizing parameterθ of the ideal objective function l(θ). In the following, we establish asymptotic consistency , similar to Theorem 1, in the parameter space. Therefore we first have to formalize the concept of closeness and optimality in the parameter space. Since a minimizing parameterθ of l(θ) may not be unique, we define Θ * = {θ * | l(θ *) ≤ l(θ) ∀θ ∈ Θ} as the set of global minimizers of l(θ), and d(θ, Θ *) = min θ * ∈Θ * {||θ − θ * || 2} as the distance of an arbitrary parameter θ to Θ *. Based on these definitions, it can be shown that Algorithm 1 is asymptotically consistent in a sense that the minimizer ofθ n,r converges almost surely to the set of optimal parameters Θ *. Theorem 2 Suppose the regularity conditions and are satisfied, h n fulfills and Θ is compact. For r > 0 and n > n 0, letθ n,r ∈ Θ be a global minimizer of the empirical objectivel almost surely. Note that Theorem 2 considers global optimizers, but equivalently holds for compact neighborhoods of a local minimum θ * (see discussion in Appendix C). After discussing the properties of noise regularization, we are interested in how to properly choose the noise intensity h, for different training data sets. Ideally, we would like to choose h so that |l | is minimized, which is practically not feasible since l(θ) is intractable. Inequality gives as an upper bound on this quantity, suggesting to minimize l 1 distance between the kernel density estimate q (h) n and the data distribution p(z). This is in turn a well-studied problem in the kernel density estimation literature (see e.g.,). Unfortunately, general solutions of this problem require knowing p(z) which is not the case in practice. Under the assumption that p(z) and the kernel function K are Gaussian, the optimal bandwidth can be derived as h = 1.06σn . In that,σ denotes the estimated standard deviation of the data, n the number of data points and d the dimensionality of Z. This formula is widely known as the rule of thumb and often used as a heuristic for choosing h. In addition, the conditions in give us further intuition. The first condition tells us that h n needs to decay towards zero as n becomes large. This reflects the general theme in machine learning that the more data is available, the less inductive bias / regularization should be imposed. The second condition suggests that the bandwidth decay must happen at a rate slower than n − 1 d. For instance, the rule of thumb fulfills these two criteria and thus constitutes a useful guideline for selecting h. However, for highly non-Gaussian data distributions, the respective h n may decay too slowly and a faster decay rate such as n − 1 1+d may be appropriate. This section provides a detailed experimental analysis of the proposed method, aiming to empirically validate the theoretical arguments outlined previously and investigating the practical efficacy of our regularization approach. In all experiments we use Gaussian pertubations of the data, i.e., K(ξ) = N (0, I). Since one of the key features of our noise regularization scheme is that it is agnostic to the choice of model, we evaluate its performance on three different neural network based CDE models: Mixture Density Networks (MDN) , Kernel Mixture Networks (KMN) and Normalizing Flows Networks (NFN) . In our experiments, we consider both simulated as well as real-world data sets. In particular, we simulate data from a 4-dimensional Gaussian Mixture (d x = 2, d y = 2) and a Skew-Normal distribution whose parameters are functionally dependent on x (d x = 1, d y = 1). In terms of real-world data, we use the following three data sources: Euro Stoxx: Daily stock-market returns of the Euro Stoxx 50 index conditioned on various stock return factors relevant in finance (d x = 14, d y = 1). UCI datasets: Standard data sets from the UCI machine learning repository . In particular, Boston Housing (The reported scores are test log-likelihoods, averaged over at least 5 random seeds alongside the respective standard deviation. For further details regarding the data sets and simulated data, we refer to Appendix E. The experiment data and code is available at TODO We complement the discussion in 4.3 with an empirical investigation of different schedules of h n . In particular, we compare a) the rule of thumb h n ∝ n − 1 4+d b) a square root decay schedule h n ∝ n − 1 1+d c) a constant bandwidth h n = const. ∈ (0, ∞) and d) no noise regularization, i.e. h n = 0. Figure 2 plots the respective test log-likelihoods against an increasing training set size n for the two simulated densities Gaussian Mixture and Skew Normal. First, we observe that bandwidth rates which conform with the decay conditions seem to converge in performance to the non-regularized maximum likelihood estimator (red) as n becomes large. This reflects the theoretical of Theorem 1. Second, a fixed bandwidth across n (green), violating, imposes asymptotic bias and thus saturates in performance vastly before its counterparts. Third, as hypothesized, the relatively slow decay of h n through the rule of thumb works better for data distributions that have larger similarities to a Gaussian, i.e., in our case the Skew Normal distribution. In contrast, the highly non-Gaussian data from the Gaussian Mixture requires faster decay rates like the square root decay schedule. Most importantly, noise regularization substantially improves the estimator's performance when only little training data is available. We now investigate how the proposed noise regularization scheme compares to classical regularization techniques. In particular, we consider an l 1 and l 2 -penalty on the neural network weights as regularization term, the weight decay technique of 1, as well a Bayesian neural network trained with variational inference using a Gaussian prior and posterior . First, we study the performance of the regularization techniques on our two simulation benchmarks. Figure 3 depicts the respective test log-likelihood across different training set sizes. For each regularization method, the regularization hyper-parameter has been optimized via grid search. As one would expect, the importance of regularization, i.e., performance difference to un-regularized model, decreases as the amount of training data becomes larger. The noise regularization scheme Table 1: Comparison of various regularization methods for three neural network based CDE models across 5 data sets. We report the test log-likelihood and its respective standard deviation (higher log-likelihood values are better). yields similar performance across the different CDE models while the other regularizers vary greatly in their performance depending on the different models. This reflects the fact that noise regularization is agnostic to the parameterization of the CDE model while regularizers in the parameter space are dependent on the internal structure of the model. Most importantly, noise regularization performs well across all models and sample sizes. In the great majority of configurations it outperforms the other methods. Especially when little training data is available, noise regularization ensures a moderate test error while the other approaches mostly fail to do so. Next, we consider real world data sets. Since now the amount of data we can use for hyper-parameter selection, training and evaluation is limited, we use 5-fold cross-validation to select the parameters for each regularization method. The test log-likelihoods, reported in Table 1, are averages over 3 different train/test splits and 5 seeds each for initializing the neural networks. The held out test set amounts to 20% of the overall data sets. Consistent with the of the simulation study, noise regularization outperforms the other methods across the great majority of data sets and CDE models. We benchmark neural network based density estimators against state-of-the art CDE approaches. While neural networks are the obvious choice when a large amount of training data is available, we pose the questions how such estimators compete against well-established non-parametric methods in small data regimes. In particular, we compare to the three following CDE methods: Conditional Kernel Density Estimation (CKDE). Non-parametric method that forms a KDE of both p(x, y) and p(x) to compute its estimate asp(y|x):=p(x, y)/p(x) (Table 2 : Comparison of conditional density estimators across 5 data sets. Reported is the test loglikelihood and its respective standard deviation (higher log-likelihood values are better). -Neighborhood kernel density estimation (NKDE). Non-parametric method that considers only a local subset of training points to form a density estimate. Semi-parametric estimator that computes the conditional density as linear combination of fixed kernels . For the kernel density estimation based methods CKDE and NKDE, we perform bandwidth selection via the rule of thumb (R.O.T) and via maximum likelihood leave-one-out cross-validation (CV-ML) . In case of LSCDE, MDN, KMN and NFN, the respective hyper-parameters are selected via 5-fold cross-validation grid search on the training set. Note that, in contrast to Section 5.2 which focuses on regularization parameters, the grid search here extends to more hyper-parameters. The respective test log-likelihood scores are listed in Table 2. For the majority of data sets, all three neural network based methods outperform all of the non-and semi-parametric methods. Perhaps surprisingly, it can be seen that, when properly regularized, neural network based CDE works well even when training data is scarce, such as in case of the Boston Housing data set. This paper addresses conditional density estimation with high-capacity models. In particular, we propose to add small random perturbations to the data during training. We demonstrate that the ing noise regularization method corresponds to a smoothness regularization and prove its asymptotic consistency. The experimental underline the effectiveness of the proposed method, demonstrating that it consistently outperforms other regularization methods across various conditional density models and data sets. This makes neural network based CDE the preferable method, even when only little training data is available. While we assess the estimator performance in terms of the test log-likelihood, an interesting question for future research is whether the noise regularization also improves the respective uncertainty estimates for downstream tasks such as safe control and decision making. Let l(D) be a loss function over a set of data points D = {z 1, ..., z N}, which can be partitioned into a sum of losses corresponding to each data point x n: Also, let each z i be perturbed by a random noise vector ξ ∼ K(ξ) with zero mean and i.i.d. elements, i.e. E ξ∼K(ξ) [ξ] = 0 and E ξ∼K(ξ) ξ n ξ j = h 2 I The ing loss l(z i + ξ) can be approximated by a second order Taylor expansion around z i Assuming that the noise ξ is small in its magnitude, O(ξ 3) may be neglected. The expected loss under K(ξ) follows directly from: Using the assumption about ξ in we can simplify as follows: In that, l(z i) is the loss without noise and we denote the elements of the column vector z. The objective function corresponding to a conditional M-projection. The sample equivalent: Corollary 1 Let Θ be a compact set and andf θ: Proof. The corollary follows directly from the uniform law of large numbers. Lemma 1 Suppose for some > 0 there exists a constant B and there exists an n 0 such that for all n > n 0 there exists a constant B almost surely. Then, the inequality where C is a constant holds with probability 1 for all n > n 0. Proof of Lemma 1 Using Hoelder's inequality and the nonnegativity of p andq Employing the regularity conditions and and writing with probability 1. Lemma 1 states regularity conditions ensuring that the expectations in l (h) n (θ) and l(θ) are wellbehaved in the limit. In particular, and imply uniform and absolute integrability of the loglikelihoods under the respective probability measures induced by p andq (h) n. Since we are interested in the asymptotic behavior, it is sufficient for to hold for n large enough with probability 1. Inequality shows that we can make |l (h) n (θ) − l(θ)| small by reducing the l 1 -distance between the true density p and the kernel density estimateq (h) n. There exists already a vast body of literature, discussing how to properly choose the kernel K and the bandwidth sequence (h n) so that |q We employ the in for our purposes, leading us to Proposition 1. Proof of Proposition 1. Let A denote the event that ∃n 0 ∀n > n 0 inequality holds for some constant C . From our regularity assumptions it follows that P(A c) = 0. Given that A holds, we just have to show that |q − − → 0. Then, the upper bound in tends to zero and we can conclude our proposition. For any δ > 0 let B n denote the event whereinq (h) n (z) is a kernel density estimate obtained based on n samples from p(z). Under the conditions in we can apply Theorem 1 of , obtaining an upper bound on the probability that does not hold, i.e. ∃u, m 0 such that P(B c n) ≤ e −un for all n > m 0. Since we need both A and B n for n → ∞ to hold, we consider the intersection of the events (A ∩ B n). Using a union bound argument it follows that ∃k 0 such that ∀n > k 0: Note that we can simply choose k 0 = max{n 0, m 0} for this to hold. Hence, e u −1 < ∞ and by the Borel-Cantelli lemma we can conclude that lim In that 1(A) denotes an indicator function which returns 1 if A is true and 0 else. Next we consider the probability that the convergence in holds for random Z (n): Note that we can dP (Z (n) ) move outside of the inner integrals, since Z (n) is independent from I and Ξ (r). Hence, we can conclude that also holds, which we denote as event A, with probability 1 for random training data. From Proposition 1 we know, that with probability 1. We denote the event that holds as B. Since P (A c) = P (B c) = 0, we can use a union bound argument to show that P (A ∩ B) = 1. From and it follows that for any n > n 0, lim with probability 1. Finally, we combine this with, obtaining that almost surely, which concludes the proof. Proof of Theorem 2. The proof follows the argument used in Theorem 1 of. In the following, we assume that holds. From Theorem 1 we know that this is the case with probability 1. Respectively, we only consider realizations of our training data Z (n) and noise samples I (r), Ξ for which the convergence in holds (see proof of Theorem 1 for details on this notation). For such realization, let (θ From the triangle inequality, it follows that for any > 0 there exists k 0 so that ∀k > k 0 given the convergence established in Theorem 1 and the continuity of l in θ. Next, the above is extended to which again holds for k large enough. This due to, the minimizer of µ i k,j k, and µ i k,j k (θ) − l(θ) < by Theorem 1. Because can be made arbitrarily small, l(θ 0) ≤ l(θ) as k → ∞. Because θ ∈ Θ is arbitrary, θ 0 must be in Θ *. In turn, since (n i) i, (r i,j) j and (i k) k, (j k) k were chosen arbitrarily, every limit point of a sequence (v i k,j k) k must be in Θ *. In the final step, we proof the theorem by contradiction. Suppose that does not hold. In this case, there must exist an > 0 and sequences However, by the previous argument the limit point of the any sequence where chosen from a set with probability mass of 1, we can conclude our proposition that lim Discussion of Theorem 2. Note that, similar to θ *,θ n,r does not have to be unique. In case there are multiple minimizers ofl (h) n,r, we can chose one of them arbitrarily and the proof of the theorem still holds. Theorem 2 considers global optimizers over a set of parameters Θ, which may not be attainable in practical settings. However, the application of the theorem to the context of local optimization is straightforward when Θ is chosen as a compact neighborhood of a local minimum θ * of l (b . In particular, the parameters of the unconditional mixture distribution p(y) are outputted by the neural network, which takes the conditional variable x as input. For our purpose, we employ a Gaussian Mixture Model (GMM) with diagonal covariance matrices as density model. The conditional density estimatep(y|x) follows as weighted sum of K Gaussianŝ wherein w k (x; θ) denote the weight, µ k (x; θ) the mean and σ 2 k (x; θ) the variance of the k-th Gaussian component. All the GMM parameters are governed by the neural network with parameters θ and input x. The mixing weights w k (x; θ) must resemble a categorical distribution, i.e. it must hold that K k=1 w k (x; θ) = 1 and w k (x; θ) ≥ 0 ∀k. To satisfy the conditions, the softmax linearity is used for the output neurons corresponding to w k (x; θ). Similarly, the standard deviations σ k (x) must be positive, which is ensured by a sofplus non-linearity. Since the component means µ k (x; θ) are not subject to such restrictions, we use a linear output layer without non-linearity for the respective output neurons. For the experiments in 5.2 and 5.1, we set K = 10 and use a neural network with two hidden layers of size 32. While MDNs resemble a purely parametric conditional density model, a closely related approach, the Kernel Mixture Network (KMN), combines both non-parametric and parametric elements . Similar to MDNs, a mixture density model ofp(y) is combined with a neural network which takes the conditional variable x as an input. However, the neural network only controls the weights of the mixture components while the component centers and scales are fixed w.r.t. to x. For each of the kernel centers, M different scale/bandwidth parameters σ m are chosen. As for MDNs, we employ Gaussians as mixture components, wherein the scale parameter directly coincides with the standard deviation. Let K be the number of kernel centers µ k and M the number of different kernel scales σ m. The KMN conditional density estimate reads as follows: As previously, the weights w k,m correspond to a softmax function. The M scale parameters σ m are learned jointly with the neural network parameters θ. The centers µ k are initially chosen by k-means clustering on the {y i} n i=1 in the training data set. Overall, the KMN model is more restrictive than MDN as the locations and scales of the mixture components are fixed during inference and cannot be controlled by the neural network. However, due to the reduced flexibility of KMNs, they are less prone to over-fit than MDNs. For the experiments in 5.2 and 5.1, we set K = 50 and M = 2. The respective neural network has two hidden layers of size 32. The Normalizing Flow Network (NFN) is similar to the MDN and KMN in that a neural network takes the conditional variable x as its input and outputs parameters for the distribution over y. For the NFN, the distribution is given by a Normalizing Flow . It works by transforming a simple base distribution and an accordingly distributed random variable Z 0 through a series of invertible, parametrized mappings f = f N • · · · • f 1 into a successively more complex distribution p(f (Z 0)). The PDF of samples z N ∼ p(f (Z 0)) can be evaluted using the change-ofvariable formula: log det ∂f n ∂z n−1 The Normalizing Flows from were introduced in the context of posterior estimation in variational inference. They are optimized for fast sampling while the likelihood evaluation for externally provided data is comparatively slow. To make them useful for CDE, we invert the direction of the flows, defining a mapping from the transformed distribution p(Z N) to the base distribution p(Z 0) by settingf −1 i (z i) = f i (z i). We experimented with three types of flows: planar flows, radial flows as parametrized by and affine flows f −1 (z) = exp(a)z +b. We have found that one affine flow combined with multiple radial flows performs favourably in most settings. For the experiments in 5.2 and 5.1, we used a standard Gaussian as the base distribution that is transformed through one affine flow and ten radial flows. The respective neural network has two hidden layers of size 32. The data generating process (x, y) ∼ p(x, y) resembles a bivariate joint-distribution, wherein x ∈ R follows a normal distribution and y ∈ R a conditional skew-normal distribution (Anděl et al., 1984). The parameters (ξ, ω, α) of the skew normal distribution are functionally dependent on x. Specifically, the functional dependencies are the following: ξ(x) = a * x + b a, b ∈ R α(x) = α low + 1 1 + e −x * (α high − α low) y ∼ SkewN ormal ξ(x), ω(x), α(x) Accordingly, the conditional probability density p(y|x) corresponds to the skew normal density function: In that, N (·) denotes the density, and Φ(·) the cumulative distribution function of the standard normal distribution. The shape parameter α(x) controls the skewness and kurtosis of the distribution. We set α low = −4 and α high = 0, giving p(y|x) a negative skewness that decreases as x increases. This distribution will allow us to evaluate the performance of the density estimators in presence of skewness, a phenomenon that we often observe in financial market variables. Figure 4a illustrates the conditional skew normal distribution. The joint distribution p(x, y) follows a Gaussian Mixture Model in R 4 with 5 Gaussian components, i.e. K = 5. We assume that x ∈ R 2 and y ∈ R 2 can be factorized, i.e. p(x, y) = When x and y can be factorized as in, the conditional density p(y|x) can be derived in closed form: wherein the mixture weights are a function of x: For details and derivations we refer the interested reader to and. The weights w k are sampled from a uniform distribution U and then normalized to sum to one. The component means are sampled from a spherical Gaussian with zero mean and standard deviation of σ = 1.5. The covariance matrices Σ y,k ) and Σ y,k ) are sampled from a Gaussian with mean 1 and standard deviation 0.5, and then projected onto the cone of positive definite matrices. Since we can hardly visualize a 4-dimensional GMM, Figure 4b depicts a 2-dimensional equivalent, generated with the procedure explained above. The goal is to predict the conditional probability density of 1-day log-returns, conditioned on 14 explanatory variables. These conditional variables comprise classical return factors from finance as well as option implied moments. For details, we refer to. Overall, the target variable is one-dimensional, i.e. y ∈ Y ⊆ R, whereas the conditional variable x constitutes a 14-dimensional vector, i.e. x ∈ X ⊆ R 14.

Represent the TLDR for finding the corresponding scientific article

A model-agnostic regularization scheme for neural network-based conditional density estimation.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.