Question: Represent the article for finding a single-sentence TLDR

Question: System identification is the process of building a mathematical model of an unknown system from measurements of its inputs and outputs. It is a key step for model-based control, estimator design, and output prediction. This work presents an algorithm for non-linear offline system identification from partial observations, i.e. situations in which the system's full-state is not directly observable. The algorithm presented, called SISL, iteratively infers the system's full state through non-linear optimization and then updates the model parameters. We test our algorithm on a simulated system of coupled Lorenz attractors, showing our algorithm's ability to identify high-dimensional systems that prove intractable for particle-based approaches. We also use SISL to identify the dynamics of an aerobatic helicopter. By augmenting the state with unobserved fluid states, we learn a model that predicts the acceleration of the helicopter better than state-of-the-art approaches. The performance of controllers and state-estimators for non-linear systems depends heavily on the quality of the model of system dynamics . System-identification tackles the problem of learning or calibrating dynamics models from data , which is often a timehistory of observations of the system and control inputs. In this work, we address the problem of learning dynamics models of partially observed, high-dimensional non-linear systems. That is, we consider situations in which the system's state cannot be inferred from a single observation, but instead requires inference over a time-series of observations. The problem of identifying systems from partial observations arises in many robotics domains (; ;). Though we often have direct measurements of a robot's pose and velocity, in many cases we cannot directly observe relevant quantities such as the temperature of actuators or the state the environment around the robot. Consider learning a dynamics model for an aerobatic helicopter. attempted to map only the helicopter's pose and velocity to its acceleration and they found their model to be inaccurate when predicting aggressive maneuvers. They posited that the substantial airflow generated by the helicopter affected the dynamics. Since it is often impossible to directly measure the state of the airflow around a vehicle, identification must be performed in a partially observed setting. System-identification is a mature field with a rich history (; 2010). Various techniques can be classified by whether they apply to linear or non-linear systems, with partially or fully observed states. Additionally, techniques are applied in an online or batch-offline setting. This work presents an approach to offline identification of non-linear and partially observed systems. When a system is fully observed, i.e. its full state is observed but corrupted by noise, a set of techniques called equation-error methods are typically employed (Åström &). In such cases, we can consider observations as independent, and minimize the error between the observed statederivatives and those predicted by the model given the control input and observed states. In partially observed settings, merely knowing the current input is insufficient to accurately predict the observation. Several black-box approaches exist to predict observations from time-series of inputs. Autoregressive approaches directly map a time-history of past inputs to observations . Recurrent neural networks and subspace-identification methods can also be used to learn blackbox dynamical systems from this data. However, in many cases prior knowledge can be used to specify structured, parameterized models of the system . Such models can be trained with less data and used with a wider array of control and state-estimation techniques than non-linear black-box models (; b; a). Techniques used to identify partially observed structured models are often based on Expectation-Maximization (EM) (; Schön et al., 2011; ;). An alternating procedure is performed in which a smoothing step uses the current system dynamics estimate to infer the distribution over state-trajectories, and is followed by a learning step that uses this distribution to update the system dynamics estimate. In the non-linear or non-Gaussian case, it is typically not possible to analytically characterize the distribution over trajectories, and thus methods based on Sequential Monte-Carlo such as Particle Smoothing (PS) (Schön et al., 2011;), or Extended Kalman Smoothing (EKS) are employed in the E-step. Though considered state-of-theart for this problem, both methods become intractable in high-dimensional state spaces. PS suffers from the curse of dimensionality, requiring an intractably large number of particles if the state space is high-dimensional , and an M-step that can be quadratic in complexity with respect to the number of particles (Schön et al., 2011). EKS-based methods are fast during the E-step, but the M-step requires approximations to integrate out state uncertainty, such as fitting non-linearities with Radial Basis Function approximators, and scales poorly with the dimension of the state-space . In this work, we present a system-identification algorithm that is suited for high-dimensional, nonlinear, and partially observed systems. By assuming that the systems are close to deterministic, as is often the case in robotics, we approximate the distribution over unobserved states using only their maximum-likelihood (ML) point-estimate. Our algorithm, called SISL (System-identification via Iterative Smoothing and Learning) performs the following two steps until convergence: • In the smoothing or E-step, we use non-linear programming to tractably find the ML pointestimate of the unobserved states. • In the learning or M-step, we use the estimate of unobserved states to improve the estimate of system parameters. The idea to use an ML point-estimate in lieu of the distribution over unobserved states in the EM procedure's E-step is not new, and, in general, does not guarantee monotonic convergence to a local optimum . However, such an approximation is equivalent to regular EM if the ML point-estimate is the only instance of unobserved variables with non-negligible probability . We apply this idea to the problem of system-identification for nearly deterministic systems, in which ML point-estimates can serve as surrogates for the true distribution over unobserved state-trajectories. The primary contribution of this work is an algorithm for identifying non-linear, partially observed systems that is able to scale to high-dimensional problems. In Section 2, we specify the assumptions underpinning our algorithm and discuss the computational methodology for using it. In Section 3, we empirically demonstrate that it is able to identify the parameters of a high-dimensional system of coupled Lorenz attractors, a problem that proves intractable for particle-based methods. We also demonstrate our algorithm on the problem of identifying the dynamics of an aerobatic helicopter, and compare against various approaches including the state-of-the-art approach . In this work, we assume that we are given a batch of trajectories containing observations y 1:T ∈ R m×T of a dynamical system as it evolves over a time horizon T, possibly forced by some known input sequence u 1:T −1. We assume that this dynamical system has a state x ∈ R n that evolves and generates observations according to the following equations, where w t is referred to as the process noise and v t as the observation noise. w t and v t are both assumed to be additive for notational simplicity, but this is not a required assumption. Without loss of generality, we can drop the dependence on u t, absorbing it into the dependence on t. We further assume that we are provided a class of parameterized models f θ (x, t) and g θ (x, t) for θ ∈ Θ that approximate the dynamical system's evolution and observation processes. The goal of our algorithm is to find the parameters θ that maximize the likelihood of the observations. That is, we seek to find: Assuming, the system is Markovian in x t, we can factorize the distributions as: In order to tractably solve the maximization problem in Equation, particle-based EM techniques typically approximate the integral as an expectation over a particle set (Schön et al., 2011;). However, since particle-based methods can struggle with high-dimensional spaces , this work seeks point estimates for the likelihoodmaximizing state sequence, which can be found using non-linear programming. Instead of solving the full maximum likelihood problem shown in Equation, we solve a surrogate problem: The maximized surrogate objective is equivalent to the original objective if the distribution over x 1:T given y 1:T and θ is well-approximated by its ML point-estimate, i.e. the distribution is a Dirac delta function. We expect this assumption to hold if the trajectories are sufficiently long, the dynamics are close to deterministic, and all relevant dynamic modes are persistently excited. Making these assumptions, we now present an algorithm for solving Equation. By taking the logarithm of the likelihood-objective in Equation, and using the factorization from Equation, we get the SISL objective J(x 1:T, θ) as follows: Jointly maximizing this objective over x 1:T and θ yields θ SISL. Though this objective can be optimized as is by a non-linear optimizer, it is not necessarily efficient to do so since θ and x 1:T are highly coupled, leading to inefficient and potentially unstable updates. For this reason, we take an iterative approach akin to EM, which performs block-coordinate ascent on the J(x 1:T, θ). At iteration k, we first perform smoothing by holding θ constant, and finding the ML point-estimate for x 1:T as follows: Here ρ x scales a soft trust-region regularizer similar to damping terms found in LevenbergMarquardt methods . The smoothing problem can be solved in O(n 2 T) by taking advantage of sparsity in the smoothing objective's Hessian matrix. In the learning step, we hold x 1:T constant and find: Again, ρ θ scales a soft trust-region regularizer, and specifying log p(θ) allows us to regularize θ toward a prior. The above optimization problem can be solved using any non-linear optimizer, such as a first or second-order gradient descent scheme. The SISL algorithm iterates between the smoothing and learning steps until convergence. The objective of our experiments is to demonstrate that the SISL algorithm is capable of identifying high-dimensional non-linear systems in partially observed settings. We do so in simulation by identifying the parameters of a system of partially observed coupled Lorenz attractors, as well as by identifying the dynamics of a real aerobatic helicopter. In the second experiment, we build on previous analysis of the dataset by attempting to characterize the interaction of the helicopter with the fluid around it, without having any direct observation of the fluid state. In this didactic experiment, we will show that: 1. SISL learns unbiased parameter estimates of systems that are close to deterministic, and, 2. SISL scales to high-dimensional problems in which particle-smoothing is intractable. To justify these claims, we use a system that is sufficiently non-linear and partially observable to make particle-based smoothing methods intractable. We choose a system of coupled Lorenz attractors for this purpose, owing to their ability to exhibit chaotic behavior and their use in non-linear atmospheric and fluid flow models (Bergé et al., 1984). Arbitrary increase in state dimensionality can be achieved by coupling multiple individual attractors. The state of a system with K coupled Lorenz attractors is x ∈ R 3K = {. . ., x 1,k, x 2,k, x 3,k, . . .}. The dynamics of the system are as follows: where H is an R 3K×3K matrix. We nominally set the parameters (σ k, ρ k, β k) to the values (10, 28, 8/3), and randomly sample the entries of H from a normal distribution to generate chaotic and coupled behavior between attractors, while avoiding self-coupling. These parameters are estimated during identification. In order to make the system partially observed, the observation y ∈ R (3K−2) is found from x as follows: where C ∈ R (3K−2)×3K is a known matrix with full row-rank, and v is the observation noise sampled from a Gaussian with diagonal covariance σ 2 v I. The entries of C are also randomly sampled from a standard normal distribution. In the following experiments, we simulate the system for T = 128 timesteps at a sample rate of ∆t = 0.04s, and integrate the system using a 4 th -order Runge-Kutta method. Initial conditions for each trajectory are sampled such that x 1,k ∼ N (−6, 2.5 2), x 2,k ∼ N (−6, 2.5 2), x 3,k ∼ N (24, 2.5 2). To test the conditions under which SISL learns unbiased parameter estimates, we simulate a single Lorenz system with H = 0, and known C ∈ R 2×3. We introduce and vary the process noise w ∼ N (0, σ 2 w I), and vary the observation noise coefficient σ v, and then attempt to estimate the parameters (σ, ρ, β). Using initial guesses within 10% of the system's true parameter values, we run SISL on a single sampled trajectory. For each choice of σ w and σ v, we repeat this process for 10 random seeds. Table 1 shows the mean and standard errors of parameter estimates for various σ w and σ v. We highlight in red the mean estimates that are not within two standard errors of their true value. We see that σ and ρ are estimated without bias for all scenarios. However, the estimate of β appears to become biased as the process noise is increased, but not as the observation noise is increased. This supports the assumption that the objective used in SISL is sound when systems evolve close to deterministically, but can be biased if it is not. State-of-the-art methods for parameter identification of partially observed systems rely on particlesmoothing (PS) to estimate a distribution over states x 1:T given a trajectory of observations y 1:T. We will experimentally demonstrate that the performance of PS does not scale to high-dimensional systems. To do so, we compare the number of particles required for PS to reliably characterize the distribution over x 1:T for a system of two and six coupled Lorenz systems. The systems are simulated with observation noise σ v = 0.01 but without process noise. We implement a PS as specified by Schön et al.. Letx t,n, w t,n be the estimated system-state and particle weight corresponding to the nth particle at the tth timestep. In order to test whether PS can reliably characterize the posterior distribution over x 1:T, we measure the weighted root mean square error (RMSE), ξ as follows: If PS is reliable using N particles, we should see that ξ N is tightly distributed across random seeds. These experimentally demonstrate that, for a sufficiently non-linear and partially observable system, the number of particles required to reliably characterize the posterior distribution over hidden states grows intractably with the dimension of the system. Since particle-based EM methods for system-identification are typically super-linear in complexity with respect to the number of particles (Schön et al., 2011), these methods are ill-suited to high-dimensional problems. To demonstrate that SISL is capable of identifying high-dimensional systems, we show that we can estimate the dynamics of an 18 dimensional system of six coupled Lorenz attractors. Moreover, as the number of trajectories provided to SISL increases, it converges to more accurate estimates. To test this claim, we sample 2, 4, and 8 trajectories from a system with parameters θ true, and σ v = 0.01. We randomly initialize each element of the parameters being optimized (θ = [σ 1:K, ρ 1:K, β 1:K, H]) to within 10% of the their value in θ true. We then run SISL on each batch, tracking the error in the estimated dynamics as training proceeds. We measure this error, which we call (θ), as follows: In the learning step, we do not regularize θ to a prior and set ρ θ = 0. In Figure 2, we see the of this experiment for four random seeds for each batch size. We can see that, as the number of trajectories used in training increases, the error in the estimated dynamics tends toward zero. Furthermore, we see that SISL convergences monotonically to a local optimum in all cases. This experiment supports our claim that SISL is able to identify the parameters of a high-dimensional, non-linear, and partially observable system that is intractable for particle-based methods. The experiments conducted thus far have demonstrated that SISL can learn unbiased parameter estimates of nearly-deterministic systems, and can scale to high-dimensional problems for which particle-based methods are intractable. In the next experiment, we use SISL to characterize the effect of unobserved states on the dynamics of an aerobatic helicopter. Characterizing the dynamics of a helicopter undergoing aggressive aerobatic maneuvers is widely considered to be a challenging system-identification problem . The primary challenge is that the forces on the helicopter depend on the induced state of the fluid around it. The state of the fluid cannot be directly observed and its dynamics model is unknown. Merely knowing the state of the helicopter and the control commands at a given time does not contain enough information to accurately predict the forces that act on it. In order to address this issue, use an approach based on Takens theorem, which suggests that a system's state can be reconstructed with a finite number of lagged-observations of it . Instead of attempting to estimate the unobserved fluid state, they directly learn a mapping from a 0.5 s long history of observed state measurements and control commands to the forces acting on the helicopter. This approach is sensible, and is equivalent to considering the past 0.5 s of observations as the system's state. However, it can require a very large number of lagged observations to represent complex phenomena. Having such a high dimensional state can make the control design and stateestimation more complicated. To avoid large input dimensions, a trade-off between the duration of the history and sample frequency is necessary. This trade-off will either hurt the resolution of low-frequency content or will alias high-frequencies. We attempt to instead explicitly model the unobserved states affecting the system. The objective of this learning problem is to predict y t, the helicopter's acceleration at time t, from an input vector u t containing the current measured state of the helicopter (its velocity and rotation rates) and the control commands. We use data collected by the Stanford Autonomous Helicopter Project . Trajectories are split into 10 s long chunks and then randomly distributed into train, test, and validation sets according to the protocol established by; and summarized in Appendix A.1. The train, test and validation sets respectively contain 466, 100 and 101 trajectories of 500 time-steps each. A simple success metric on a given trajectory is the root mean squared prediction error, RMSE =, where y (measured) t is the measured force from the dataset, y (pred) t is the force predicted by the model, and T is the number of time-steps in each trajectory. We first consider a naive baseline that does not attempt to account for the time-varying nature of the fluid-state. We train a neural-network to map only the current helicopter state and control commands to the accelerations: y t = NN θn (u t), where NN θn is a neural-network with parameters θ n. We refer to this model as the naive model. We also compare to the work of. They predict y t using a time-history u t−H:t of H lagged observations of the helicopter's measured state and control commands. This input is passed through a ReLU-activated neural network with a single hidden-layer combined with what they call a Quadratic Lag Model. As a baseline, we reproduce their performance with a single deep neural network y t = NN θ h (u t−H:t) with parameters θ h. We call this neural network model the H25 model. Both of these models can be trained via stochastic gradient descent to minimize the Mean-Squared-Error (MSE) of their predictions for y. The optimization methodology for these models is described in Appendix A.2. As a third baseline, we compare with subspace-identification methods . We letỹ t = y t − NN θn (u t) be the prediction errors of the trained naive model. We use the MATLAB command n4sid to fit a linear dynamical system of the following form: Here, x ∈ R d is the unobserved state with arbitrary dimension d. The learned parameters are We use a state dimension of 10 and call this model the SID model. The n4sid algorithm scales super-linearly with the amount of data supplied, and thus we train on 10 randomly sampled subsets of 100 trajectories each, and report the distribution in prediction performance. Particle-based EM methods are not presented as baselines because they are intractable on problems with large state-spaces, as shown in Section 3.1.2. Similar to the parameterization used for subpace-identification, we fit the prediction errors of the naive model using the following dynamical system: where NNθ NL is a neural network, and θ NL = [A NL, B NL, C NL, D NL,θ NL] are the learned parameters. While learning, we assume that both process and observation noise are distributed with diagonal Gaussian covariance matrices σ w I and σ v I respectively. The values of σ w and σ v are treated as hyperparmeters of SISL. Here as well, we use a state dimension of 10 and call this model the NL model. The optimization methodology for this model is described in Appendix A.2. It should be noted that the system we learn need not actually correspond to an interpretable model of the fluid-state, but only of time-varying hidden-states that are useful for predicting the accelerations of the helicopter. Expert knowledge of helicopter aerodynamics could be used to further inform a gray-box model trained with SISL. The test RMSE of the naive and H25 models can be evaluated directly on the test trajectories using next-step prediction. However, the SID and NL models require an estimate of the unobserved state before making a prediction. The natural analog of next-step prediction is extended Kalman filtering (EKF), during which states are recursively predicted and corrected given observations. At a given time-step, a prediction ofỹ t is made using the current estimate of x t, and is used in the computation of RMSE. The state-estimate is then corrected with the measuredỹ t. Figure 3a shows the RMSE of the compared models on trajectories in the test-set. We see that the NL model is able to consistently predict the accelerations on the helicopter with better accuracy than any of the other models. The naive model performs on average 2.9 times worse than the H25 model, and its can be found in Appendix A.3. The SID model notably outperforms the state-of-the-art H25 model, suggesting that a large linear dynamical system can be used to approximate a non-linear and partially observable system (Korda & Mezić, 2018). However, introducing non-linearity as in the NL model noticeably improves performance. Figure 3b depicts the errors in prediction over a sample trajectory in the test-set. Here, we also see that the NL model is able to attenuate the time-varying error present in predictions made by the H25, suggesting that it has accurately characterized the dynamics of unobserved, time-varying states. This experiment validates the effectiveness of SISL to identify a non-linear dynamical model of unobserved states that affect the forces acting an aerobatic helicopter. This paper presented an algorithm for system identification of non-linear systems given partial state observations. The algorithm optimizes system parameters given a time history of observations by iteratively finding the most likely state-history, and then using it to optimize the system parameters. The approach is particularly well suited for high-dimensional and nearly deterministic problems. In simulated experiments on a partially observed system of coupled Lorenz attractors, we showed that our algorithm can perform identification on a problem that particle-based EM methods are fundamentally ill-suited for. We also validated that our algorithm is an effective replacement for identification methods based on EM if the system is close to deterministic, but can yield biased parameter estimates if it is not. We then used our algorithm to model the time-varying hiddenstates that affect the dynamics of an aerobatic helicopter. Our approach outperforms state-of-the-art methods because it is able to fit large non-linear models to unobserved states. We aim to apply our algorithm to system identification problems in a number of domains. There has recently been interest in characterizing the dynamics of aircraft with high aspect ratios, for which the difficult-to-observe bending modes substantially impact dynamics. Additionally, the inability to measure friction forces in dynamic interactions involving contact typically stands in the way of system identification, and thus requires algorithms that are capable of identification under partial observation. A APPENDIX In this work we use the dataset gathered by and available at http://heli. stanford.edu/. A gas-powered helicopter was flown by a professional pilot to collect a large dataset of 6290s of flight. There are four controls: the longitudinal and lateral cyclic pitch, the tail rotor pitch and the collective pitch. The state is measured thanks to an accelerometer, a gyroscope, a magnetometer and vision cameras. provide the raw data, as well as states estimates in the Earth reference frame obtained with extended Kalman smoothing.'s protocol, we use the fused sensor data and downsample it from 100Hz to 50Hz. From the Earth frame accelerations provided in the dataset, we compute body frame accelerations (minus gyroscopic terms) which are the prediction targets for our training. Using the notations from , we can write the helicopter dynamics in the following form: where s ∈ R 13 is the helicopter state consisting of its position r, quaternion-attitude q, linear velocity v, angular velocity ω, and δ ∈ R 4 to be the control command. C 12 is the rotation-matrix from the body to Earth reference frame, and f v and f ω are the linear and angular accelerations caused by aerodynamic forces, and are what we aim to predict. This notation connects with the one used in our paper in the following way: • We define u as the concatenation of all inputs to the model, including the relevant state variables v and ω and control commands δ. • We define y as the output predicted, which would correspond to a concatenation of f v and f ω. • We define x as the vector of unobserved flow states to be estimated and is not present in their model. Neural-networks in the naive and H25 models have eight hidden layers of size 32 each, and tanh non-linearities. We optimize these models using an Adam optimizer with a harmonic learning rate decay, and mini-batch size of 512. The neural network in the NL model has two hidden layers of size 32 each, and tanh non-linearity. We train the NL model with SISL, using ρ x = ρ θ = 0.5, σ w = σ v = 1.0, and use an Adam optimizer to optimize Equation in the learning step. The learning rate for dynamics parameters in θ NL is 5.0 × 10 −4 and observation parameters in θ NL is 1.0 × 10 −3. For its relative robustness, we optimize Equation using a non-linear least squares optimizer with a Trust-Region Reflective algorithm (-) in the smoothing step. This step can be solved very efficiently by providing the solver with the block diagonal sparsity pattern of the Jacobian matrix. To evaluate the test metric, running an EKF is required. The output of an EKF depends on several user-provided parameters: • x 0: value of the initial state • Σ 0: covariance of error on initial state • Q: covariance of process noise • R: covariance of observation noise In this work, we assume that Q, R and Σ 0 are all set to the identity matrix. x 0 is assumed to be 0 on all dimensions. A well-tuned EKF with an inaccurate initial state value converges to accurate estimations in only a few time steps of transient behavior. Since the H25 model needs 25 past inputs to predict its first output prediction, we drop the first 25 predictions from the EKF when computing RMSE, thereby omitting some of the transient regime.

Represent the short summary for finding the corresponding scientific article (long):

This work presents a scalable algorithm for non-linear offline system identification from partial observations.

Represent the TLDR for finding the corresponding scientific article

We introduce two approaches for efficient and scalable inference in stochastic simulators for which the density cannot be evaluated directly due to, for example, rejection sampling loops.


Question: Represent the natural language

Question: Various gradient compression schemes have been proposed to mitigate the communication cost in distributed training of large scale machine learning models. Sign-based methods, such as signSGD , have recently been gaining popularity because of their simple compression rule and connection to adaptive gradient methods, like ADAM. In this paper, we perform a general analysis of sign-based methods for non-convex optimization. Our analysis is built on intuitive bounds on success probabilities and does not rely on special noise distributions nor on the boundedness of the variance of stochastic gradients. Extending the theory to distributed setting within a parameter server framework, we assure exponentially fast variance reduction with respect to number of nodes, maintaining 1-bit compression in both directions and using small mini-batch sizes. We validate our theoretical findings experimentally. One of the key factors behind the success of modern machine learning models is the availability of large amounts of training data (; ;). However, the state-of-the-art deep learning models deployed in industry typically rely on datasets too large to fit the memory of a single computer, and hence the training data is typically split and stored across a number of compute nodes capable of working in parallel. Training such models then amounts to solving optimization problems of the form where f m: R d → R represents the non-convex loss of a deep learning model parameterized by x ∈ R d associated with data stored on node m. Arguably, stochastic gradient descent (SGD) (; ;) in of its many variants (; ; ; ;) is the most popular algorithm for solving. In its basic implementation, all workers m ∈ {1, 2, . . ., M} in parallel compute a random approximation g m (x k) of ∇f m (x k), known as the stochastic gradient. These approximations are then sent to a master node which performs the aggregation The aggregated vector is subsequently broadcast back to the nodes, each of which performs an update of the form x k+1 = x k − γ kĝ (x k), thus updating their local copies of the parameters of the model. Typically, communication of the local gradient estimatorsĝ m (x k) to the master forms the bottleneck of such a system (; ;). In an attempt to alleviate this communication bottleneck, a number of compression schemes for gradient updates have been proposed and analyzed;; signSGD, Theorem 1 Step size Weak noise assumptions? ρi > ). A compression scheme is a (possibly randomized) mapping Q: R d → R d, applied by the nodes toĝ m (x k) (and possibly also by the master to aggregated update in situations when broadcasting is expensive as well) in order to reduce the number of bits of the communicated message. Sign-based compression. Although most of the existing theory is limited to unbiased compression schemes, i.e., on operators Q satisfying EQ(x) = x, biased schemes such as those based on communicating signs of the update entries only often perform much better (; ; ; ; ; ; ;). The simplest among these sign-based methods is signSGD (see also Algorithm 1; Option 1), whose update direction is assembled from the component-wise signs of the stochastic gradient. Adaptive methods. While ADAM is one of the most popular adaptive optimization methods used in deep learning , there are issues with its convergence and generalization properties. It was noted in that the behaviour of ADAM is similar to a momentum version of signSGD. Connection between sign-based and adaptive methods has long history, originating at least in Rprop and RMSprop . Therefore, investigating the behavior of signSGD can improve our understanding on the convergence of adaptive methods such as ADAM. We now summarize the main contributions of this work. Our key are summarized in Table 1. 1 In fact, bounded variance assumption, being weaker than bounded second moment assumption, is stronger (or, to be strict, more curtain) than SPB assumption in the sense of differential entropy, but not in the direct sense. The entropy of probability distribution under the bounded variance assumption is bounded, while under the SPB assumption it could be arbitrarily large. This observation is followed by the fact that for continuous random variables, the Gaussian distribution has the maximum differential entropy for a given variance (see https://en.wikipedia.org/wiki/Differential_entropy). • 2 methods for 1-node setup. In the M = 1 case, we study two general classes of sign based methods for minimizing a smooth non-convex function f. The first method has the standard form while the second has a new form not considered in the literature before: • Key novelty. The key novelty of our methods is in a substantial relaxation of the requirements that need to be imposed on the gradient estimatorĝ(x k) of the true gradient ∇f (x k). In sharp contrast with existing approaches, we allowĝ(x k) to be biased. Remarkably, we only need one additional and rather weak assumption onĝ(x k) for the methods to provably converge: we require the signs of the entries ofĝ(x k) to be equal to the signs of the entries of ∇f (x k) with a probability strictly larger than 1 /2 (see Section 2; Assumption 1). We show through a counterexample (see Section 2.2) that this assumption is necessary. • Geometry. As a byproduct of our analysis, we uncover a mixed l 1 -l 2 geometry of sign descent methods (see Section 3). • Convergence theory. We perform a complexity analysis of methods and (see Section 4.1; Theorem 1). While our complexity bounds have the same O(1 / √ K) dependence on the number of iterations, they have a better dependence on the smoothness parameters associated with f. Theorem 1 is the first on signSGD for non-convex functions which does not rely on mini-batching, and which allows for step sizes independent of the total number of iterations K. Finally, Theorem 1 in can be recovered from our general Theorem 1. Our bounds are cast in terms of a novel norm-like function, which we call the ρ-norm, which is a weighted l 1 norm with positive variable weights. • Distributed setup. We extend our to the distributed setting with arbitrary M (Section 4.2), where we also consider sign-based compression of the aggregated gradients. In this section we describe our key (and weak) assumption on the gradient estimatorĝ(x) of the true gradient ∇f (x), and give an example which shows that without this assumption, method can fail. Assumption 1 (SPB: Success Probability Bounds). For any x ∈ R d, we have access to an independent (and not necessarily unbiased) estimatorĝ(x) of the true gradient g(x):= ∇f (x) that satisfies for all x ∈ R d and all i ∈ {1, 2, . . ., d}. We will refer to the probabilities ρ i as success probabilities. As we will see, they play a central role in the convergence of sign based methods. We stress that Assumption 1 is the only assumption on gradient noise in this paper. Moreover, we argue that it is reasonable to require from the sign of stochastic gradient to show true gradient direction more likely than the opposite one. Extreme cases of this assumption are the absence of gradient noise, in which case ρ i = 1, and an overly noisy stochastic gradient, in which case ρ i ≈ 1 2. Remark 1. Assumption 1 can be relaxed by replacing bounds with However, if Prob(signĝ i (x) = 0) = 0 (e.g. in the case ofĝ i (x) has continuous distributions), then these two bounds are identical. Under review as a conference paper at ICLR 2020 Extension to stochastic sign oracle. Notice that we do not requireĝ to be unbiased. Moreover, we do not assume uniform boundedness of the variance, or of the second moment. This observation allows to extend existing theory to more general sign-based methods with a stochastic sign oracle. By a stochastic sign oracle we mean an oracle that takes x k ∈ R d as an input, and outputs a random vectorŝ k ∈ R d with entries in ±1. However, for the sake of simplicity, in the rest of the paper we will work with the signSGD formulation, i.e., we letŝ k = signĝ(x k). Here we analyze a counterexample to signSGD discussed in. Consider the following least-squares problem with unique minimizer x * =:, where ε ∈ and stochastic gradientĝ(x) = ∇ a i, x 2 = 2 a i, x a i with probabilities 1/2 for i = 1, 2. Let us take any point from the line l = {(z 1, z 2): z 1 + z 2 = 2} as initial point x 0 for the algorithm and notice that signĝ(x) = ±(1, −1) for any x ∈ l. Therefore, signSGD with any step-size sequence remains stuck along the line l, whereas the problem has a unique minimizer at the origin. We now investigate the cause of the divergence. In this counterexample, Assumption 1 is violated. Indeed, note that for i = 1, 2. By S:= {x ∈ R 2 : a 1, x · a 2, x > 0} = ∅ denote the open cone of points having either an acute or an obtuse angle with both a i' s. Then for any x ∈ S, the sign of the stochastic gradient is ±(1, −1) with probabilities 1 /2. Hence for any x ∈ S, we have low success probabilities: So, in this case we have an entire conic region with low success probabilities, which clearly violates. Furthermore, if we take a point from the complement open coneS c, then the sign of stochastic gradient equals to the sign of gradient, which is perpendicular to the axis of S (thus in the next step of the iteration we get closer to S). For example, if a 1, x < 0 and a 2, x > 0, then signĝ(x) = (1, −1) with probability 1, in which case x − γ signĝ(x) gets closer to low success probability region S. In summary, in this counterexample there is a conic region where the sign of the stochastic gradient is useless (or behaves adversarially), and for any point outside that region, moving direction (which is the opposite of the sign of gradient) leads toward that conic region. To justify our SPB assumption, we show that it holds under general assumptions on gradient noise. Lemma 1 (see B.1). Assume that for any point x ∈ R d, we have access to an independent and unbiased estimatorĝ(x) of the true gradient g(x). Assume further that each coordinateĝ i has a unimodal and symmetric distribution with variance σ Next, we remove the distribution condition and add a strong growth condition together with fixed mini-batch size. Lemma 2 (see B.2). Assume that for any point x ∈ R d, we have access to an independent, unbiased estimatorĝ(x) of the true gradient g(x), with coordinate-wise bounded variances σ for some constant c. Then, choosing a mini-batch size τ > 2c, we get Finally, we give an adaptive condition on mini-batch size for the SPB assumption to hold. Lemma 3 (see B.3). Assume that for any point x ∈ R d we have access to an independent and unbiased estimatorĝ(x) of the true gradient g(x). Let σ 2 i = σ 2 i (x) be the variance and ν Under review as a conference paper at ICLR 2020 In this section we introduce the concept of a norm-like function, which call ρ-norm, induced from success probabilities. Used to measure gradients in our convergence rates, ρ-norm is a technical tool enabling the analysis. be the collection of probability functions from the SPB assumption. We define the ρ-norm of gradient Note that ρ-norm is not a norm as it may not satisfy the triangle inequality. However, under SPB assumption, ρ-norm is positive definite as it is a weighted l 1 norm with positive (and variable) weights, and g ρ = 0 if and only if g = 0. Under the assumptions of Lemma 2, ρ-norm can be lower bounded by a weighted l 1 norm with positive constant weights 1 − 2c Under the assumptions of Lemma 1, ρ-norm can be lower bounded by a mixture of the l 1 and squared l 2 norms: Note that l 1,2 -norm is again not a norm. However, it is positive definite, continuous and order preserving, i.e., for any g k, g,g ∈ R d we have: i) g l 1,2 ≥ 0 and g l 1,2 = 0 if and only if g = 0;. From these three properties it follows that g k l 1,2 → 0 implies g k → 0. These properties are important as we will measure convergence rate in terms of the l 1,2 norm in the case of unimodal and symmetric noise assumption. To understand the nature of the l 1,2 norm, consider the following two cases when σ i (x) ≤ c|g i (x)| +c for some constants c,c ≥ 0. If the iterations are in ε-neighbourhood of a minimizer x * with respect to the l ∞ norm (i.e., max 1≤i≤d |g i | ≤ ε), then the l 1,2 norm is equivalent to scaled l 2 norm squared: On the other hand, if iterations are away from a minimizer (i.e., min 1≤i≤d |g i | ≥ L), then the l 1,2 -norm is equivalent to scaled l 1 norm: g 1. These equivalences are visible in Figure 1, where we plot the level sets of g → g l 1,2 at various distances from the origin. Similar mixed norm observation was also noted in. Now we turn to our theoretical of sign based methods. First we give our general convergence under the SPB assumption. Afterwards, we present convergence in the distributed setting under the unimodal and symmetric noise assumptions. Throughout the paper we assume that f: and is L-smooth with some non-negative constants That is, we assume that We allow f to be nonconvex. We now state our convergence for Algorithm 1 under the general SPB assumption. Under review as a conference paper at ICLR 2020 Theorem 1 (Non-convex convergence of signSGD, see B.4). Under the SPB assumption, signSGD (Algorithm 1 with Option 1) with step sizes γ k = γ 0 / √ k + 1 converges as follows If γ k ≡ γ > 0, we get 1 /K convergence to a neighbourhood of the solution: We now comment on the above : • Generalization. Theorem 1 is the first general on signSGD for non-convex functions without mini-batching, and with step sizes independent of the total number of iterations K. Known convergence (; on signSGD use mini-batches and/or step sizes dependent on K. Moreover, they also use unbiasedness and unimodal symmetric noise assumptions, which are stronger assumptions than our SPB assumption (see Lemma 1). Finally, Theorem 1 in can be recovered from Theorem 1 (see Section D for the details). • Convergence rate. Rates and can be arbitrarily slow, depending on the probabilities ρ i. This is to be expected. At one extreme, if the gradient noise was completely random, i.e., if ρ i ≡ 1/2, then the ρ-norm would become identical zero for any gradient vector and rates would be trivial inequalities, leading to divergence as in the counterexample. At other extreme, if there was no gradient noise, i.e., if ρ i ≡ 1, then the ρ-norm would be just the l 1 norm and from we get the rateÕ(1/ √ K) with respect to the l 1 norm. However, if we know that ρ i > 1/2, then we can ensure that the method will eventually converge. • Geometry. The presence of the ρ-norm in these rates suggests that there is no particular geometry (e.g., l 1 or l 2) associated with signSGD. Instead, the geometry is induced from the success probabilities. For example, in the case of unbiased and unimodal symmetric noise, the geometry is described by the mixture norm l 1,2. • Practicality. The rate (as well as) supports the common learning schedule practice of using a constant step size for a period of time, and then halving the step-size and continuing this process. For a reader interested in comparing Theorem 1 with a standard for SGD, we state the standard in the Section C. We now state a general convergence rate for Algorithm 1 with Option 2. Theorem 2 (see B.5). Under the SPB assumption, Algorithm 1 (Option 2) with step sizes γ k = γ 0 / √ k + 1 converges as follows: In the case of constant step size γ k = γ > 0, the same rate as is achieved. Comparing Theorem 2 with Theorem 1, notice that a small modification in Algorithm 1 can remove the log-dependent factor from; we then bound the average of past gradient norms instead of the minimum. On the other hand, in a big data regime, function evaluations in Algorithm 1 (Option 2, line 4) are infeasible. Clearly, Option 2 is useful only when one can afford function evaluations and has rough estimates about the gradients (i.e., signs of stochastic gradients). This option should be considered within the framework of derivative-free optimization. In this part we present the convergence of distributed signSGD (Algorithm 2) with majority vote introduced in. Majority vote is considered within a parameter server framework, where for each coordinate parameter server receives one sign from each node and sends Under review as a conference paper at ICLR 2020 back the sign sent by the majority of nodes. Known convergence (; use O(K) mini-batch size as well as O(1/K) constant step size. In the sequel we remove this limitations extending Theorem 1 to distributed training. In distributed setting the number of nodes M get involved in geometry introducing new ρ M -norm, which is defined by the regularized incomplete beta function I (see B.6). Now we can state the convergence rate of distributed signSGD with majority vote. Theorem 3 (Non-convex convergence of distributed signSGD, see B.6). Under SPB assumption, distributed signSGD (Algorithm 2) with step sizes γ k = γ 0 / √ k + 1 converges as follows For constant step sizes γ k ≡ γ > 0, we have convergence up to a level proportional to step size γ: Variance Reduction. Using Hoeffding's inequality, we show that 1, where ρ(x) = min 1≤i≤d ρ i (x) > 1 /2. Hence, in some sense, we have exponential variance reduction in terms of number of nodes (see B.7). Number of nodes. Notice that theoretically there is no difference between 2l−1 and 2l nodes, and this in not a limitation of the analysis. Indeed, as it is shown in the proof, expected sign vector at the master with M = 2l − 1 nodes is the same as with M = 2l nodes: E sign(ĝ is the sum of stochastic sign vectors aggregated from nodes. The intuition behind this phenomenon is that majority vote with even number of nodes, e.g. M = 2l, fails to provide any sign Under review as a conference paper at ICLR 2020 with little probability (it is the probability of half nodes voting for +1, and half nodes voting for −1). However, if we remove one node, e.g. M = 2l − 1, then master receives one sign-vote less but gets rid of that little probability of failing the vote (sum of odd number of ±1 cannot vanish). So, somehow this two things cancel each other and we gain no improvement in expectation adding one more node to parameter server framework with odd number of nodes. We verify our theoretical experimentally using the MNIST dataset with feed-forward neural network (FNN) and the well known Rosenbrock (non-convex) function with d = 10 variables: Stochastic formulation of minimization problem for Rosenbrock function is as follows: at any point x ∈ R d we have access to biased stochastic gradientĝ(x) = ∇f i (x) + ξ, where index i is chosen uniformly at random from {1, 2, . . ., d − 1} and ξ ∼ N (0, ν 2 I) with ν > 0. Figure 2 illustrates the effect of multiple nodes in distributed training with majority vote. As we see increasing the number of nodes improves the convergence rate. It also supports the claim that in expectation there is no improvement from 2l − 1 nodes to 2l nodes. Figure 4 shows the robustness of SPB assumption in the convergence rate with constant step size. We exploited four levels of noise in each column to demonstrate the correlation between success probabilities and convergence rate. In the first experiment (first column) SPB assumption is violated strongly and the corresponding rate shows divergence. In the second column, probabilities still violating SPB assumption are close to the threshold and the rate shows oscillations. Next columns show the improvement in rates when success probabilities are pushed to be close to 1. Under review as a conference paper at ICLR 2020 Figure 5: Performance of signSGD with variable step size (γ 0 = 0.25) under four different noise levels (mini-batch size 1, 2, 5, 7) using Rosenbrock function. As in the experiments of Figure 4 with constant step size, these plots show the relationship between success probabilities and the convergence rate. In low success probability regime (first and second columns) we observe oscillations, while in high success probability regime (third and forth columns) oscillations are mitigated substantially. Under review as a conference paper at ICLR 2020 to a neighborhood of the solution. We fixed gradient noise level by setting mini-batch size 2 and altered the constant step size. For the first column we set bigger step size γ = 0.25 to detect the divergence (as we slightly violated SPB assumption). Then for the second and third columns we set γ = 0.1 and γ = 0.05 to expose the convergence to a neighborhood of the minimizer. For the forth column we set even smaller step size γ = 0.01 to observe a slower convergence. Here we state the well-known Gauss's inequality on unimodal distributions 3. Theorem 4 (Gauss's inequality). Let X be a unimodal random variable with mode m, and let σ 2 m be the expected value of (X − m) 2. Then for any positive value of r, Applying this inequality on unimodal and symmetric distributions, direct algebraic manipulations give the following bound: where m = µ and σ 2 m = σ 2 are the mean and variance of unimodal, symmetric random variable X, and r ≥ 0. Now, using the assumption that eachĝ i (x) has unimodal and symmetric distribution, we apply this bound for and get a bound for success probabilities Improvment on Lemma 1 and l 1,2 norm: The bound after Gauss inequality can be improved including a second order term Hence, continuing the proof of Lemma 1, we get and we could have defined l 1,2 -norm in a bit more complicated form as Under review as a conference paper at ICLR 2020 B.2 SUFFICIENT CONDITIONS FOR SPB: PROOF OF LEMMA 2 Letĝ (τ) be the gradient estimator with mini-batch size τ. It is known that the variance forĝ (τ) is dropped by at least a factor of τ, i.e. Hence, estimating the failure probabilities of signĝ (τ) when g i = 0, we have which imples We will split the derivation into three lemmas providing some intuition on the way. The first two lemmas establish success probability bounds in terms of mini-batch size. Essentially, we present two methods: one works well in the case of small randomness, while the other one in the case of non-small randomness. In the third lemma, we combine those two bounds to get the condition on mini-batch size ensuring SPB assumption. Lemma 4. Let X 1, X 2,..., X τ be i.i.d. random variables with non-zero mean µ:= EX 1 = 0, finite variance σ 2:= E|X 1 − µ| 2 < ∞. Then for any mini-batch size τ ≥ 1 Proof. Without loss of generality, we assume µ > 0. Then, after some adjustments, the proof follows from the Chebyshev's inequality: where in the last step we used independence of random variables X 1, X 2,..., X τ. Obviously, bound is not optimal for big variance as it becomes a trivial inequality. In the case of non-small randomness a better bound is achievable additionally assuming the finitness of 3th central moment. Lemma 5. Let X 1, X 2,..., X τ be i.i.d. random variables with non-zero mean µ:= EX 1 = 0, positive variance σ 2:= E|X 1 − µ| 2 > 0 and finite 3th central moment ν 3:= E|X 1 − µ| 3 < ∞. Then for any mini-batch size τ ≥ 1 where error function erf is defined as Proof. Again, without loss of generality, we may assume that µ > 0. Informally, the proof goes as follows. As we have an average of i.i.d. random variables, we approximate it (in the sense of distribution) by normal distribution using the Central Limit Theorem (CLT). Then we compute success probabilities for normal distribution with the error function erf. Finally, we take into account the approximation error in CLT, from which the third term with negative sign appears. More formally, we apply Berry-Esseen inequality 4 on the rate of approximation in CLT : where N ∼ N has the standard normal distribution. Setting t = −µ √ τ /σ, we get It remains to compute the second probability using the cumulative distribution function of normal distribuition and express it in terms of the error function: Clearly, bound is better than when randomness is high. On the other hand, bound is not optimal for small randomness (σ ≈ 0). Indeed, one can show that in a small randomness regime, while both variance σ 2 and third moment ν 3 are small, the ration ν/σ might blow up to infinity producing trivial inequality. For instance, taking X i ∼ Bernoulli(p) and letting p → 1 gives ν/σ = O (1 − p) − 1 /6. This behaviour stems from the fact that we are using CLT: less randomness implies slower rate of approximation in CLT. As a of these two bounds on success probabilities, we conclude a condition on mini-batch size for the SPB assumption to hold. Under review as a conference paper at ICLR 2020 Lemma 6. Let X 1, X 2,..., X τ be i.i.d. random variables with non-zero mean µ = 0 and finite variance σ 2 < ∞. Then where ν 3 is (possibly infinite) 3th central moment. Proof. First, if σ = 0 then the lemma holds trivially. If ν = ∞, then it follows immediately from Lemma 4. Assume both σ and ν are positive and finite. In case of τ > 2σ 2 /µ 2 we apply Lemma 4 again. Consider the case τ ≤ 2σ 2 /µ 2, which implies we get which together with gives Hence, SPB assumption holds if It remains to show that erf Lemma follows from Lemma applying it to i.i.d. dataĝ First, from L-smoothness assumption we have where g k = g(x k),ĝ k =ĝ(x k),ĝ k,i is the i-th component ofĝ k andL is the average value of L i' s. Taking conditional expectation given current iteration x k gives Under review as a conference paper at ICLR 2020 Using the definition of success probabilities ρ i we get Plugging this into and taking full expectation, we get Therefore Now, in case of decreasing step sizes where we have used the following standard inequalities In the case of constant step size γ k = γ Under review as a conference paper at ICLR 2020 B.5 CONVERGENCE ANALYSIS: PROOF OF THEOREM 2 Clearly, the iterations {x k} k≥0 of Algorithm 1 (Option 2) do not increase the function value in any iteration, i.e. E[f (x k+1)|x k ] ≤ f (x k). Continuing the proof of Theorem 1 from, we get where we have used the following inequality The proof for constant step size is the same as in Theorem 1. The proof of Theorem 3 goes with the same steps as in Theorem 1, except the derivation- is replaced by where we have used the following lemma. Lemma 7. Assume that for some point x ∈ R d and some coordinate i ∈ {1, 2, . . ., d}, master node receives M independent stochastic signs signĝ m i (x), m = 1,..., M of true gradient g i (x) = 0. Let g (M) (x) be the sum of stochastic signs aggregated from nodes: where l = [(M +1) /2] and ρ i > 1 /2 is the success probablity for coordinate i.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

General analysis of sign-based methods (e.g. signSGD) for non-convex optimization, built on intuitive bounds on success probabilities.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the scientific article to find a short TLDR

Question: Off-policy learning, the task of evaluating and improving policies using historic data collected from a logging policy, is important because on-policy evaluation is usually expensive and has adverse impacts. One of the major challenge of off-policy learning is to derive counterfactual estimators that also has low variance and thus low generalization error. In this work, inspired by learning bounds for importance sampling problems, we present a new counterfactual learning principle for off-policy learning with bandit feedbacks. Our method regularizes the generalization error by minimizing the distribution divergence between the logging policy and the new policy, and removes the need for iterating through all training samples to compute sample variance regularization in prior work. With neural network policies, our end-to-end training algorithms using variational divergence minimization showed significant improvement over conventional baseline algorithms and is also consistent with our theoretical . Off-policy learning refers to evaluating and improving a deterministic policy using historic data collected from a stationary policy, which is important because in real-world scenarios on-policy evaluation is oftentimes expensive and has adverse impacts. For instance, evaluating a new treatment option, a clinical policy, by administering it to patients requires rigorous human clinical trials, in which patients are exposed to risks of serious side effects. As another example, an online advertising A/B testing can incur high cost for advertisers and bring them few gains. Therefore, we need to utilize historic data to perform off-policy evaluation and learning that can enable safe exploration of the hypothesis space of policies before deploying them. There has been extensive studies on off-policy learning in the context of reinforcement learning and contextual bandits, including various methods such as Q learning BID33 ), doubly robust estimator BID8 ), self-normalized (Swaminathan & Joachims (2015b) ), etc. A recently emerging direction of off-policy learning involves the use of logged interaction data with bandit feedback. However, in this setting, we can only observe limited feedback, often in the form of a scalar reward or loss, for every action; a larger amount of information about other possibilities is never revealed, such as what reward we could have obtained had we taken another action, the best action we should have take, and the relationship between the change in policy and the change in reward. For example, after an item is suggested to a user by an online recommendation system, although we can observe the user's subsequent interactions with this particular item, we cannot anticipate the user's reaction to other items that could have been the better options. Using historic data to perform off-policy learning in bandit feedback case faces a common challenge in counterfactual inference: How do we handle the distribution mismatch between the logging policy and a new policy and the induced generalization error? To answer this question, BID34 derived the new counterfactual risk minimization framework, that added the sample variance as a regularization term into conventional empirical risk minimization objective. However, the parametrization of policies in their work as linear stochastic models has limited representation power, and the computation of sample variance regularization requires iterating through all training samples. Although a first-order approximation technique was proposed in the paper, deriving accurate and efficient end-to-end training algorithms under this framework still remains a challenging task. Our contribution in this paper is three-fold:1. By drawing a connection to the generalization error bound of importance sampling BID6 ), we propose a new learning principle for off-policy learning with bandit feedback. We explicitly regularize the generalization error of the new policy by minimizing the distribution divergence between it and the logging policy. The proposed learning objective automatically trade off between emipircal risk and sample variance. 2. To enable end-to-end training, we propose to parametrize the policy as a neural network, and solves the divergence minimization problem using recent work on variational divergence minimization BID26 ) and Gumbel soft-max BID18 ) sampling. 3. Our experiment evaluation on benchmark datasets shows significant improvement in performance over conventional baselines, and case studies also corroborates the soundness of our theoretical proofs. We first review the framework of off-policy learning with logged bandit feedback introduced in BID34. A policy maps an input x ∈ X to a structured (discrete) output y ∈ Y. For example, the input x can be profiles of users, and we recommend movies of relevance to the users as the output y; or in the reinforcement learning setting, the input is the trajectory of the agent, and the output is the action the agent should take in the next time point. We use a family of stochastic policies, where each policy defines a posterior distribution over the output space given the input x, parametrized by some θ, i.e., h θ (Y|x). Note that here a distribution which has all its probability density mass on one action corresponds to a deterministic policy. With the distribution h(Y|x), we take actions by sampling from it, and each action y has a probability of h(y|x) being selected. In the discussion later, we will use h and h(y|x) interchangeably when there will not create any confusion. In online systems, we observe feedbacks δ(x, y; y *) for the action y sampled from h(Y|x) by comparing it to some underlying'best' y * that was not revealed to the system. For example, in recommendation system, we can use a scalar loss function δ(x, y; y DISPLAYFORM0 with smaller values indicating higher satisfaction with recommended items. The expected risk of a policy h(Y|x) is defined as DISPLAYFORM1, and the goal of off-policy learning is to find a policy with minimum expected risk on test data. In the off-line logged learning setting, we only have data collected from a logging policy h 0 (Y|x), and we aim to find an improved policy h(Y|x) that has lower expected risks R(h) < R(h 0). Specifically, the data we will use will be DISPLAYFORM2, where δ i and p i are the observed loss feedback and the logging probability (also called propensity score), and N is the number of training samples. Two main challenges are associated with this task: 1) If the distribution of a logging policy is skewed towards a specific region of the whole space, and doesn't have support everywhere, feedbacks of certain actions cannot be obtained and improvement for these actions is not possible as a . 2) since we cannot compute the expectation exactly, we need to resort to empirical estimation using finite samples, which creates generalization error and needs additional regularization. A vanilla approach to solve the problem is propensity scoring approach using importance sampling BID28 ), by accounting for the distribution mismatch between h and h 0. Specifically, we can rewrite the expected risk w.r.t h as the risk w.r.t h 0 using an importance reweighting: DISPLAYFORM3 With the collected historic dataset D, we can estimate the empirical riskR D (h), short asR(h) DISPLAYFORM4 2.2 COUNTERFACTUAL RISK MINIMIZATION BID34 pointed out several flaws with the vanilla approach, namely, not being invariant to loss scaling, large and potentially unbounded variance. To regularize the variance, the authors proposed a regularization term for sample variance derived from empirical Bernstein bounds. The modified objective function to minimize is now: DISPLAYFORM5, where DISPLAYFORM6 u i is the average of {u i} obtained from training data, and V ar(ū) is the sample variance of {u i}.As the variance term is dependent on the whole dataset, stochastic training is difficult, the authors approximated the regularization term via first-order Taylor expansion and obtained a stochastic optimization algorithm. Despite its simplicity, such first-order approximation neglects the non-linear terms from second-order and above, and introduces approximation errors while trying to reduce the sample variance. Instead of estimating variance empirically from the samples, which prohibits direct stochastic training, the fact that we have a parametrized version of the policy h(Y|x) motivates us to think: can we derive a variance bound directly from the parametrized distribution?We first note that the empirical risk termR(h) is the average loss reweigthed by importance sampling function DISPLAYFORM0 h0(y|x), and a general learning bound exist for importance sampling weights. Let z be a random variable and the importance sampling weight w(z) = p(z) p0(z), where p and p 0 are two probability density functions, the following identity holds Lemma 1. BID6 ) For a random variable z, let p(z) and p 0 (z) be two distribuion density function defined for z, and l(z) be a loss function of z bounded in. Let w = w(z) = p(z)/p 0 (z) be the importance sampling weight, the following identity holds: DISPLAYFORM1, where D 2 is the Rényi divergence D α BID27 ) with α = 2, i.e. squared Chi-2 divergence. Based on this lemma, we can derive an upper bound for the second moment of the weighted loss Theorem 1. Let X be a random variable distributed according to distribution P with density p(x), Y be a random variable, and δ(x, y) be a loss function over DISPLAYFORM2 For two sampling distributions of y, h(y|x) and h 0 (y|x), define their conditional divergence as d 2 (h(y|x)||h 0 (y|x); P(x)), we have DISPLAYFORM3 The bound is similar to Eq. with the difference that we are now working with a joint distribution over x, y. Detailed proofs can be found in Appendix 1.From the above theorem, we are able to derive a generalization bound between the expected risk R(h) and empirical riskR(h) using the distribution divergence function as Theorem 2. Let R h be the expected risk of the new policy on loss function δ, andR h be the emprical risk. We additionally assume the divergence is bounded by DISPLAYFORM4 Then with probability at least 1 − η, DISPLAYFORM5 The proof of this theorem is an application of Bernstein inequality and the second moment bound, and detailed proof is in Appendix 7. This highlights the bias-variance trade-offs as seen in empirical risk minimization (ERM) problems, whereR h approximates the emipircal risk/ bias, and the third term characterize the variance of the solution with distribution divergence (Recall V ar(w) = d 2 (h||h 0) − 1). It thus motivates us that in bandit learning setting, instead of directly optimizing the reweighed loss and suffer huge variance in test setting, we can try to minimize the variance regularized objectives as DISPLAYFORM6 λ = 2L 2 log 1/η is a model hyper-parameter controlling the trade-off between empirical risk and model variance, but we are still faced with the challenge of setting λ empirically and the difficuty in optimizing the objective (See Appendix for a comparison). Thus, in light of the recent success of distributionally robust learning, we explore an alternative formulation of the above regularized ERM in the next subsection. Instead of solving a'loss + regularizer' objective function, we here study a closely related constrained optimizationf formulation, whose intuition comes from the method of Langaragian mutliplier for constrained optimization. The new formulation is: DISPLAYFORM0, where ρ is a pre-determined constant as the regularization hyper-parameter. By applying Theorem, for a policy h, we have DISPLAYFORM1 This inequality shows that the robust objectiveR d(h||h0≤ρ) (h) is also a good surrogate of the true risk R(h), with their difference bouned by the regularization hyper-parameter ρ and approaches 0 when N → ∞.At first glance, the new objective function removes the needs to compute the sample variance in existing bounds, but when we have a parametrized distribution of h(y|x), and finite samples DISPLAYFORM2, estimating the divergence function is not an easy task. In the next subsection, we will present how recent f-gan networks for variational divergence minimization BID26 ) and Gumbel soft-max sampling BID18 ) can help solve the task. Discussion: Possibility of Counterfactual Learning: One interesting aspect of our bounds also stresses the need for the stochasticity of the logging policy BID21 ). For a deterministic logging policy, if the corresponding probability distribution can only have some peaked masses, and zeros elsewhere in its domain, our intution suggests that learning will be difficult, as those regions are never explored. Our theory well reflects this intuition in the calculation of the divergence term, the integral of form y h 2 (y|x)/h 0 (y|x)dy. A deterministic policy has a non-zero measure region of h 0 (Y|x) with probability density of h 0 (y|x) = 0, while the corresponding h(y|x) can have finite values in the region. The ing integral is thus unbounded, and in turn induces an unbounded generalization bound, making counterfactual learning in this case not possible. The derived variance regularized objective requires us to minimize the square root of the condi- DISPLAYFORM0 dy. For simplicity, we can examine the term inside the expectation operation first. With simple calculation, we have DISPLAYFORM1, where f (t) = t 2 − 1 is a convex function in the domain {t : t ≥ 0} with f = 0. Combining with the expectation operator gives a minimization objective of D f (h||h 0 ; P(X)) (+1 omitted as constant).The above calculation draws connection between our divergence and the f-divergence measure BID25 ). Follow the f-GAN for variational divergence minimization method proposed in BID26, we can reach a lower bound of the above objective as DISPLAYFORM2 For the second equality, as f is a convex function and applying Fenchel convex duality (f * = sup u {u v − f (u)}) gives the dual formulation. Because the expectation is taken w.r.t to x while the supreme is taken w.r.t. all functions T, we can safely swap the two operators. We note that the bound is tight when T 0 (x) = f (h/h 0), where f is the first order derivative of f as f (t) = 2t BID25 ).The third inequality follows because we restrict T to a family of functions instead of all functions. Luckily, the universal approximation theorem of neural networks BID15 states that neural networks with arbitrary number of hidden units can approximate continous functions on a compact set with any desired precision. Thus, by choosing the family of T to be the family of neural networks, the equality condition of the second equality can be satisfied theoretically. The final objective is a saddle point of a function T (x, y): X × Y → R that maps input pairs to a scalar value, and the policy we want to learn h(Y|x) acts as a sampling distribution. Although being a lower bound with achievable equality conditions, theoretically, this saddle point trained with mini-batch estimation is a consistent estimator of the true divergence. We use D f = sup T T dhdx − f * (T)dh 0 dx to denote the true divergence, and DISPLAYFORM3 dx the empirical estimator we use, whereĥ and h 0 are the emipircal distribution obtained by sampling from the two distribution respectively. DISPLAYFORM4 Proof. Let's start by decomposing the estimation error. DISPLAYFORM5, where the first term of error comes from restricting the parametric family of T to a family of neural networks, and the second term of error involves the approximation error of an emipirical mean estimation to the true distribution. By the universal approximation theorem, we have e 0 = 0, and that ∃T ∈ T, such that T = T 0.For the second term e 1, we plug in T 0 and have it rewritten as DISPLAYFORM6 For the first term, DISPLAYFORM7 we can see that this is the diffrence between an empirical distribution and the underlying population distribution. We can verify that the strong law of large numbers (SLLN) applies. By optimality condition, T 0 = h(y|x) h0(y|x), where both h and h 0 are probability density functions. By the bounded loss assumption, the ratio is integrable. Similarly, f * (T 0) = 2T 0 − 1 is also integrable. Thus, we can apply SLLN and conclude the term → 0. For the second term, we can apply Theroem 5 from [] and also obtain it → a.s.0.Again, a generative-adversarial approach BID12 ) can be applied. Toward this end, we represent the T function as a discriminator network parametrized as T w (x, y). We then parametrize the distribution of our policy h(y|x) as another generator neural network h θ (y|x) mapping x to the probability of sampling y. For structured output problems with discrete values of y, to allow the gradients of samples obtained from sampling backpropagated to all other parameters, we use the Gumbel soft-max sampling BID18 ) methods for differential sampling from the distribution h(y|x). We list the complete training procedure Alg. 1 for completeness. DISPLAYFORM8 sampled from logging policy h 0; a predefined threshold D 0; an initial generator distribution h θ 0 (y|x); an initial discriminator function T w 0 (x, y); max iteration I Result: An optimized generator h θ * (y|x) distribution that has minimum divergence to h 0 initialization; whileD f (h||h 0 ; P(X)) > D 0 or iter < I do Sample a mini-batch'real' samples (x i, y i) from D; Sample a mini-batch x from D, and construct'fake' samples (x i,ŷ i) by samplingŷ from h θ t (y|x) with Gubmel soft-max; Update DISPLAYFORM9 For our purpose of minimizing the variance regularization term, we can similarly derive a training algorithm, as the gradient of t → √ t + 1 can also be backpropagated. With the above two components, we are now ready to present the full treatment of our end-to-end learning for counterfactual risk minimization from logged data. The following algorithm solve the robust regularized formulation and for completeness, training for the original ERM formulation in Sec. 3.1 (referred to co-training version in the later experiment sections) is included in Appendix 7. DISPLAYFORM0 sampled from h 0; regularization hyper-parameter ρ, and maximum iteration of divergence minimization steps I, and max epochs for the whole algorithm M AX Result: An optimized generator h * θ (y|x) that is an approximate minimizer of R(w) initialization; while epoch < M AX do / * Update θ to minimize the reweighted loss * / Sample a mini-batch of m samples from D; Update θ t+1 = θ t − η θ g 1; / * Update discriminator and generator for divergence minimization * / Call Algorithm 1 to minimize the divergence D 2 (h||h 0 ; P(X)) with threshold = ρ, and max iter set to I; end Algorithm 2: Minimizing Variance Regularized Risk -Separate TrainingThe algorithm works in two seperate training steps: 1) update the parameters of the policy h to minimize the reweighed loss 2) update the parameters of the policy/ generator and the discriminator to regularize the variance thus to improve the generalization performance of the new policy. Exploiting historic data is an important problem in multi-armed bandit and its variants such as contextual bandit and has wide applications BID31; BID30; BID4 ). Approaches such as doubly robust estimators BID8 ) have been proposed, and recent theoretical study explored the finite-time minimax risk lower bound of the problem ), and an adaptive learning algorithm using the theoretical analysis. Bandits problems can be interpreted as a single-state reinforcement learning (RL) problems, and techniques including doubly robust estimators BID19;; BID24 ) have also been extended to RL domains. Conventional techniques such as Q function learning, and temporal difference learning BID33 ) are alternatives for off-policy learning in RL by accounting for the Markov property of the decision process. Recent works in deep RL studies have also addressed off-policy updates by methods such as multi-step bootstrapping (Mahmood et al. FORMULA3), off-policy training of Q functions BID14 ).Learning from logs traces backs to BID16 and BID28, where propensity scores are applied to evaluate candidate policies. In statistics, the problem is also described as treatment effect estimation (Imbens FORMULA4), where the focus is to estimate the effect of an intervention from observational studies that are collected by a different intervention. BID5 derived unbiased counterfactual estimators to study an example of computational advertising; another set of techniques reduce the bandit learning to a weighted supervised learning problems (Zadrozny et al. FORMULA4), but is shown to have poor generalization performance BID4 ).Although our variance regularization aims at off-policy learning with bandit feedback, part of the proof comes from the study of generalization bounds in importance sampling problems BID6 ), where the original purpose was to account for the distribution mismatch between training data and testing distribution, also called covariate shift, in supervised learning. BID7 also discussed variance regularized empirical risk minimization for supervised learning with a convex objective function, which has connections to distributionally robust optimization problem BID2 ). It will be of further interest to study how our divergence minimization technique can be applied to supervised learning and domain adaptation BID32; BID13 ) problems as an alternative to address the distribution match issue. Regularization for our objective function has close connection to the distributionally robust optimization techniques BID3 ), where instead of minizing the emiprical risk to learn a classifier, we minimize the supreme the emipirical risk over an ellipsoid uncertainty set. Wasserstein distance between emipircal distribution and test distribution is one of the most well studied contraint and is proven to achieve robust generalization performance (Esfahani & Kuhn ( For empirical evaluation of our proposed algorithms, we follow the conversion from supervised learning to bandit feedback method BID0). For a given supervised dataset DISPLAYFORM0, we first construct a logging policy h 0 (Y|x), and then for each sample x i, we sample a prediction y i ∼ h 0 (y|x i), and collect the feedback as δ(y * i, y i). For the purpose of benchmarks, we also use the conditional random field (CRF) policy trained on 5% of D * as the logging policy h 0, and use hamming loss, the number of incorrectly misclassified labels between y i and y * i, as the loss function δ BID34 ). To create bandit feedback datasets D = {x i, y i, δ i, p i}, each of the samples x i were passed four times to the logging policy h 0 and sampled actions y i were recorded along with the loss value δ i and the propensity score p i = h 0 (y i |x i).In evaluation, we use two type of evaluation metrics for the probabilistic policy h(Y|x). The first is the expected loss (referred to as 'EXP' later) R(h) = 1 Ntest i E y∼h(y|xi) δ(y * i, y), a direct measure of the generalization performance of the learned policy. The second is the average hamming loss of maximum a posteriori probability (MAP) prediction y MAP = arg max h(y|x) derived from the learned policy, as MAP is a faster way to generate predictions without the need for sampling in practice. However, since MAP predictions only depend on the regions with highest probability, and doesn't take into account the diverse of predictions, two policies with same MAP performance could have very different generalization performance. Thus, a model with high MAP performance but low EXP performance might be over-fitting, as it may be centering most of its probability masses in the regions where h 0 policy obtained good performance. Baselines Vanilla importance sampling algorithms using inverse propensity score (IPS), and the counterfactual risk minimization algorithm from Swaminathan & Joachims (2015a) (POEM) are compared, with both L-BFGS optimization and stochastic optimization solvers. The hyperparameters are selected by performance on validation set and more details of their methods can be found in the original paper BID34 ).Neural network policies without divergence regularization (short as "NN-NoReg" in later discussions) is also compared as baselines, to verify the effectiveness of variance regularization. Dataset We use four multi-label classification dataset collected in the UCI machine learning repo BID1 ), and perform the supervised to bandit conversion. We report the statistics in TAB2 in the Appendix. For these datasets, we choose a three-layer feed-forward neural network for our policy distribution, and a two or three layer feed-forward neural network as the discriminator for divergence minimization. Detailed configurations can be found in the Appendix 7.For benchmark comparison, we use the separate training version 2 as it has faster convergence and better performance (See Sec. 6.5 for an empirical comparison). The networks are trained with Adam BID20 ) of learning rate 0.001 and 0.01 respectively for the reweighted loss and the divergence minimization part. We used PyTorch to implement the pipelines and trained networks with Nvidia K80 GPU cards. Codes for reproducing the as well as preprocessed data can be downloaded with the link 1Results by an average of 10 experiment runs are obtained and we report the two evaluation metrics in TAB0. We report the regularized neural network policies with two Gumbel-softmax sampling schemes, soft Gumbel soft-max (NN-Soft), and straight-through Gumbel soft-max (NN-Hard).As we can see from the , by introducing a neural network parametrization of the polices, we are able to improve the test performance by a large margin compared to the baseline CRF policies, as the representation power of networks are often reported to be stronger than other models. The introduction of additional variance regularization term (comparing NN-Hard/Soft to NN-NoReg), we can observe an additional improvement in both testing loss and MAP prediction loss. We observe no significant difference between the two Gumbel soft-max sampling schemes. To study the effectiveness of variance regularization quantitatively, we vary the maximum number of iterations (I in Alg. 2) we take in each divergence minimization sub loop. For example,'NNHard-10' indicates that we use ST Gubmel soft-max and set the maximum number of iterations to 10. Here we set the thresholds for divergence slightly larger so maximum iterations are executed so that are more comparable. We plot the expected loss in test sets against the epochs average over 10 runs with error bars using the dataset yeast. As we can see from the figure, models with no regularization (gray lines in the figure) have higher loss, and slower convergence rate. As the number of maximum iterations for divergence minimization increases, the test loss decreased faster and the final test loss is also lower. This behavior suggests that by adding the regularization term, our learned policies are able to generalize better to test sets, and the stronger the regularization we impose by taking more divergence minimization steps, the better the test performance is. The regularization also helps the training algorithm to converge faster, as shown by the trend. Our theoretical bounds implies that the generalization performance of our algorithm improves as the number of training samples increases. We vary the number of passes of training data x was passed to the logging policy to sample an action y, and vary it in the range 2 [1,2,...,8] with log scales. When the number of training samples in the bandit dataset increases, both models with and without regularization have an increasing test performance in the expected loss and reaches a relatively stable level in the end. Moreover, regularized policies have a better generalization performance compared to the model without regularization constantly. This matches our theoretical intuitions that explicitly regularizing the variance can help improve the generalization ability, and that stronger regularization induces better generalization performance. But as indicated by the MAP performance, after the replay of training samples are more than 2 4, MAP prediction performance starts to decrease, which suggests the models may be starting over-fitting already. In this section, we use some experiments to present the difference in two training schemes: cotraining in Alg. 3 and the easier version Alg. 2. For the second algorithm, we also compare the two Gumbel-softmax sampling schemes in addition, denoted as Gumbel-softmax, and Straight-Through (ST) Gumbel-softmax respectively. The figures suggest that blending the weighted loss and distribution divergence performs slightly better than the model without regularization, however, the training is much more difficult compared to the separate training scheme, as it's hard to balance the gradient of the two parts of the objective function. We also observe no significant performance difference between the two sampling schemes of the Gumbel-softmax. In this section, we discuss how the effect of logging policies, in terms of stochasticity and quality, will affect the learning performance and additional visualizations of other metrics can be found in the Appendix 7.As discussed before, the ability of our algorithm to learn an improved policy relies on the stochasticity of the logging policy. To test how this stochasticity affects our learning, we modify the parameter of h 0 by introducing a temperature multiplier α. For CRF logging policies, the prediction is made by normalizing values of w T φ(x, y), where w is the model parameter and can be modified by α with w → αw. As α becomes higher, h 0 will have a more peaked distribution, and ultimately become a deterministic policy with α → ∞.We varied α in the range of 2 [−1,1,...,8], and report the average ratio of expected test loss to the logging policy loss of our algorithms (Y-axis in Fig 4a, where smaller values indicate a larger improvement). We can see that NN polices are performing better than logging policy when the stochasticity of h 0 is sufficient, while after the temperature parameter increases greater than 2 3, it's much harder and even impossible (ratio ¿ 1) to learn improved NN policies. We also note here that the stochasticity doesn't affect the expected loss values themselves, and the drop in the ratios mainly ed from the decreased loss of the logging policy h 0. In addition, comparing within NN policies, policies with stronger regularization have slight better performance against models with weaker ones, which in some extent shows the robustness of our learning principle. The decreasing stochasticity of h 0 makes it harder to obtain an improved NN policy, and our regularization can help the model be more robust and achieve better generalization performance. b) As h 0 improves, the models constantly outperform the baselines, however, the difficulty is increasing with the quality of h 0. Note: more visualizations of other metrics can be found in the appendix 7.Finally, we discusses the impact of logging policies to the our learned improved policies. Intuitively, a better policy that has lower hamming loss can produce bandit datasets with more correct predictions, however, it's also possible that the sampling biases introduced by the logging policy is larger, and such that some predictions might not be available for feedbacks. To study the trade-off between better policy accuracy and the sampling biases, we vary the proportion of training data points used to train the logging policy from 0.05 to 1, and compare the performance of our improved policies obtained by in Fig. 4b. We can see that as the logging policy improves gradually, both NN and NN-Reg policies are outperforming the logging policy, indicating that they are able to address the sampling biases. The increasing ratios of test expected loss to h 0 performance, as a proxy for relative policy improvement, also matches our intuition that h 0 with better quality is harder to beat. In this paper, we started from an intuition that explicitly regularizing variance can help improve the generalization performance of off-policy learning for logged bandit datasets, and proposed a new training principle inspired by learning bounds for importance sampling problems. The theoretical discussion guided us to a training objective as the combination of importance reweighted loss and a regularization term of distribution divergence measuring the distribution match between the logging policy and the policy we are learning. By applying variational divergence minimization and Gumbel soft-max sampling techniques, we are able to train neural network policies end-to-end to minimize the variance regularized objective. Evaluations on benchmark datasets proved the effectiveness of our learning principle and training algorithm, and further case studies also verified our theoretical discussion. Limitations of the work mainly lies in the need for the propensity scores (the probability an action is taken by the logging policy), which may not always be available. Learning to estimate propensity scores and plug the estimation into our training framework will increase the applicability of our algorithms. For example, as suggested by BID6, directly learning importance weights (the ratio between new policy probability to the logging policy probability) has comparable theoretical guarantees, which might be a good extension for the proposed algorithm. Although the work focuses on off-policy from logged data, the techniques and theorems may be extended to general supervised learning and reinforcement learning. It will be interesting to study how A. PROOFS DISPLAYFORM0 We apply Lemma 1 to z, importance sampling weight function w(z) = p(z)/p 0 (z) = h(y|x)/h 0 (y|x), and loss l(z)/L, we have DISPLAYFORM1 Thus, we have DISPLAYFORM2 Proof. For a single hypothesis denoted as δ with values DISPLAYFORM3 By Lemma 1, the variance can be bounded using Reni divergence as DISPLAYFORM4 Applying Bernstein's concentration bounds we have DISPLAYFORM5 σ 2 (Z)+ LM/3 ), we can obtain that with probability at least 1 − η, the following bounds for importance sampling of bandit learning holds DISPLAYFORM6, where the second inequality comes from the fact that DISPLAYFORM7 sampled from logging policy h 0; regularization hyper-parameter λ Result: An optimized generator h * θ (y|x) that is an approximate minimizer of R(w) initialization; while Not Converged do / * Update discriminator * / Sample a mini-batch of'fake' samples (x i,ŷ i) with x i from D andŷ i ∼ h θ t (y|x i); Sample a mini-batch of'real' samples (x i, y i) from D; Update w t+1 = w t + η w ∂F (T w, h θ); / * Update generator * / Sample a mini-batch of m samples from D; Sample a mini-batch of m 1'fake' samples; Estimate the generator gradient as g 2 = F (T w, h θ); Update θ t+1 = θ t − η θ (g 1 + λg 2); end Algorithm 3: Minimizing Variance Regularized Risk -Co-Training Version We report the statistics of the datasets as in the following table. For the latter two datasets TMC, (c) The effect of stochasticity of h0 vs ratio of test loss with MAP Figure 5: As the logging policy becomes more deterministic, NN policies are still able to find improvement over h 0 in a) expected loss and b) loss with MAP predictions. c) We cannot observe a clear trend in terms of the performance of MAP predictions. We hypothesize it from that h 0 policy already has good MAP prediction performance by centering some of the masses. While NN policies can easily pick up the patterns, it will be difficult to beat the baselines. We believe this phenomenon worth further investigation. (c) The quality of h0 vs ratio of expected test loss with MAP Figure 6: a) As the quality of the logging policy increases, NN policies are still able to find improvement over h 0 in expected loss. and b) c) For MAP predictions, however, it will be really difficult for NN policies to beat if the logging policy was already exposed to full training data and trained in a supervised fashion.

represent the tldr for finding the corresponding scientific article

For off-policy learning with bandit feedbacks, we propose a new variance regularized counterfactual learning algorithm, which has both theoretical foundations and superior empirical performance.

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO