Question: In this paper, we propose to combine imitation and reinforcement learning via the idea of reward shaping using an oracle. We study the effectiveness of the near- optimal cost-to-go oracle on the planning horizon and demonstrate that the cost- to-go oracle shortens the learner’s planning horizon as function of its accuracy: a globally optimal oracle can shorten the planning horizon to one, leading to a one- step greedy Markov Decision Process which is much easier to optimize, while an oracle that is far away from the optimality requires planning over a longer horizon to achieve near-optimal performance. Hence our new insight bridges the gap and interpolates between imitation learning and reinforcement learning. Motivated by the above mentioned insights, we propose Truncated HORizon Policy Search (THOR), a method that focuses on searching for policies that maximize the total reshaped reward over a finite planning horizon when the oracle is sub-optimal. We experimentally demonstrate that a gradient-based implementation of THOR can achieve superior performance compared to RL baselines and IL baselines even when the oracle is sub-optimal. Reinforcement Learning (RL), equipped with modern deep learning techniques, has dramatically advanced the state-of-the-art in challenging sequential decision problems including high-dimensional robotics control tasks as well as video and board games BID13 BID23. However, these approaches typically require a large amount of training data and computational resources to succeed. In response to these challenges, researchers have explored strategies for making RL more efficient by leveraging additional information to guide the learning process. Imitation learning (IL) is one such approach. In IL, the learner can reference expert demonstrations BID0, or can access a cost-to-go oracle BID19, providing additional information about the long-term effects of learner decisions. Through these strategies, imitation learning lowers sample complexity by reducing random global exploration. For example, BID25 shows that, with access to an optimal expert, imitation learning can exponentially lower sample complexity compared to pure RL approaches. Experimentally, researchers also have demonstrated sample efficiency by leveraging expert demonstrations by adding demonstrations into a replay buffer BID28 BID14, or mixing the policy gradient with a behavioral cloning-related gradient BID18.Although imitating experts can speed up the learning process in RL tasks, the performance of the learned policies are generally limited to the performance of the expert, which is often sub-optimal in practice. Previous imitation learning approaches with strong theoretical guarantees such as Data Aggregation (DAgger) BID20 and Aggregation with Values (AGGREVATE) BID19 can only guarantee a policy which performs as well as the expert policy or a one-step deviation improvement over the expert policy.1 Unfortunately, this implies that imitation learning with a sub-optimal expert will often return a sub-optimal policy. Ideally, we want the best of both IL and RL: we want to use the expert to quickly learn a reasonable policy by imitation, while also exploring how to improve upon the expert with RL. This would allow the learner to overcome the sample inefficiencies inherent in a pure RL strategy while also allowing the learner to eventually surpass a potentially sub-optimal expert. Combining RL and IL is, in fact, not new. BID5 attempted to combine IL and RL by stochastically interleaving incremental RL and IL updates. By doing so, the learned policy will either perform as well as the expert policy-the property of IL BID19, or eventually reach a local optimal policy-the property of policy iteration-based RL approaches. Although, when the expert policy is sub-optimal, the learned locally optimal policy could potentially perform better than the expert policy, it is still difficult to precisely quantify how much the learner can improve over the expert. In this work, we propose a novel way of combining IL and RL through the idea of Reward Shaping BID16. Throughout our paper we use cost instead of reward, and we refer to the concept of reward shaping with costs as cost shaping. We assume access to a cost-to-go oracle that provides an estimate of expert cost-to-go during training. The key idea is that the cost-to-go oracle can serve as a potential function for cost shaping. For example, consider a task modeled by a Markov Decision Process (MDP). Cost shaping with the cost-to-go oracle produces a new MDP with an optimal policy that is equivalent to the optimal policy of the original MDP BID16. The idea of cost shaping naturally suggests a strategy for IL: pick a favourite RL algorithm and run it on the new MDP reshaped using expert's cost-to-go oracle. In fact, BID16 demonstrated that running SARSA BID26 on an MDP reshaped with a potential function that approximates the optimal policy's value-to-go, is an effective strategy. We take this idea one step further and study the effectiveness of the cost shaping with the expert's cost-to-go oracle, with a focus on the setting where we only have an imperfect estimatorV e of the cost-to-go of some expert policy π e, i.e.,V e = V *, where V * is the optimal policy's cost-to-go in the original MDP. We show that cost shaping with the cost-to-go oracle shortens the learner's planning horizon as a function of the accuracy of the oracleV e compared to V *. Consider two extremes. On one hand, when we reshape the cost of the original MDP with V * (i.e.,V e = V *), the reshaped MDP has an effective planning horizon of one: a policy that minimizes the one-step cost of the reshaped MDP is in fact the optimal policy (hence the optimal policy of the original MDP). On the other hand, when the cost-to-go oracle provides no information regarding V *, we have no choice but simply optimize the reshaped MDP (or just the original MDP) using RL over the entire planning horizon. With the above insight, we propose the high-level strategy for combining IL and RL, which we name Truncated HORizon Policy Search with cost-to-go oracle (THOR). The idea is to first shape the cost using the expert's cost-to-go oracleV e, and then truncate the planning horizon of the new MDP and search for a policy that optimizes over the truncated planning horizon. For discrete MDPs, we mathematically formulate this strategy and guarantee that we will find a policy that performs better than the expert with a gap that can be exactly quantified (which is missing in the previous work of BID5). In practice, we propose a gradient-based algorithm that is motivated from this insight. The practical algorithm allows us to leverage complex function approximators to represent policies and can be applied to continuous state and action spaces. We verify our approach on several MDPs with continuous state and action spaces and show that THOR can be much more sample efficient than strong RL baselines (we compared to Trust Region Policy Optimization with Generalized Advantage Estimation (TRPO-GAE) ), and can learn a significantly better policy than AGGREVATE (we compared to the policy gradient version of AGGREVATE from BID25) with access only to an imperfect cost-to-go oracle. Previous work has shown that truncating the planning horizon can in a tradeoff between accuracy and computational complexity. BID8 proposed a model-based RL approach that focuses on a search for policies that maximize a sum of k-step rewards with a termination value that approximates the optimal value-to-go. Their algorithm focuses on the model-based setting and the discrete state and action setting, as the algorithm needs to perform k-step value iteration to compute the policy. Another use of the truncated planning horizon is to trade off bias and variance. When the oracle is an approximation of the value function of the agent's current policy, by using k-step rollouts bottomed up by the oracle's return, truncating the planning horizon trades off bias and variance of the estimated reward-to-go. The bias-variance tradeoff has been extensively studied in Temporal Difference Learning literature BID27 and policy iteration literature as well BID9 BID15 is perhaps the closest to our work. In Theorem 5 in the Appendix of Ng's dissertation, Ng considers the setting where the potential function for reward shaping is close to the optimal value function and suggests that if one performs reward shaping with the potential function, then one can decrease the discount factor of the original MDP without losing the optimality that much. Although in this work we consider truncating the planning steps directly, Theorem 5 in Ng's dissertation and our work both essentially considers trading off between the hardness of the reshaped MDP (the shorter the planning horizon, the easier the MDP to optimize) and optimality of the learned policy. In addition to this tradeoff, our work suggests a path toward understanding previous imitation learning approaches through reward shaping, and tries to unify IL and RL by varying the planning horizon from 1 to infinity, based on how close the expert oracle is to the optimal value function. Another contribution of our work is a lower bound analysis that shows that performance limitation of AGGREVATE with an imperfect oracle, which is missing in previous work BID19. The last contribution of our work is a model-free, actor-critic style algorithm that can be used for continuous state and action spaces. We consider the problem of optimizing Markov Decision Process defined as M 0 = (S, A, P, C, γ). Here, S is a set of S states and A is a set of A actions; P is the transition dynamics at such that for any s ∈ S, s ∈ S, a ∈ A, P (s |s, a) is the probability of transitioning to state s from state s by taking action a. For notation simplicity, in the rest of the paper, we will use short notation P sa to represent the distribution P (·|s, a). The cost for a given pair of s and a is c(s, a), which is sampled from the cost distribution C(s, a) with mean valuec(s, a). A stationary stochastic policy π(a|s) computes the probability of generating action a given state s. The value function V π M0 and the state action cost-to-go Q π M0,h (s, a) of π on M 0 are defined as: DISPLAYFORM0 where the expectation is taken with respect to the randomness of M 0 and the stochastic policy π. DISPLAYFORM1 The objective is to search for the optimal policy π * such that π * = arg min π V π (s), ∀s ∈ S.Throughout this work, we assume access to an cost-to-go oracleV e (s): S → R. Note that we do not requireV e (s) to be equal to V * M0. For example,V e (s) could be obtained by learning from trajectories demonstrated by the expert π e (e.g., Temporal Difference Learning ), orV e could be computed by near-optimal search algorithms via access to ground truth information BID7 BID5 BID24 or via access to a simulator using Dynamic Programming (DP) techniques BID6 BID17. In our experiment, we focus on the setting where we learn aV e (s) using TD methods from a set of expert demonstrations. Given the original MDP M 0 and any potential functions Φ: S → R, we can reshape the cost c(s, a) sampled from C(s, a) to be: DISPLAYFORM0 Denote the new MDP M as the MDP obtained by replacing c by c in M 0: M = (S, A, P, c, γ). BID16 showed that the optimal policy π * M on M and the optimal policy π * M0 on the original MDP are the same: π * M (s) = π * M0 (s), ∀s. In other words, if we can successfully find π * M on M, then we also find π * M0, the optimal policy on the original MDP M 0 that we ultimately want to optimize. In IL, when given a cost-to-go oracle V e, we can use it as a potential function for cost shaping.. As cost shaping does not change the optimal policy, we can rephrase the original policy search problem using the shaped cost: DISPLAYFORM0 for all s ∈ S. Though Eq. 2 provides an alternative objective for policy search, it could be as hard as the original problem as DISPLAYFORM1, which can be easily verified using the definition of cost shaping and a telescoping sum trick. As directly optimizing Eq 2 is as difficult as policy search in the original MDP, previous IL algorithms such as AGGREVATE essentially ignore temporal correlations between states and actions along the planning horizon and directly perform a policy iteration over the expert policy at every state, i.e., they are greedy with respect to A e asπ(s) = arg min a A e (s, a), ∀s ∈ S. The policy iteration theorem guarantees that such a greedy policyπ performs at least as well as the expert. Hence, when the expert is optimal, the greedy policyπ is guaranteed to be optimal. However when V e is not the optimal value function, the greedy policyπ over A e is a one-step deviation improvement over the expert but is not guaranteed to be close to the optimal π *. We analyze in detail how poor the policy ing from such a greedy policy improvement method could be when V e is far away from the optimal value function in Sec. 3. In this section we study the dependency of effective planning horizon on the cost-to-go oracle. We focus on the setting where we have access to an oracleV e (s) which approximates the cost-to-go of some expert policy π e (e.g., V e could be designed by domain knowledge BID16 or learned from a set of expert demonstrations). We assume the oracle is close to V * M0, but imperfect: |V e − V * M0 | = for some ∈ R +. We first show that with such an imperfect oracle, previous IL algorithms AGGREVATE and AGGREVATE D BID19 BID25 are only guaranteed to learn a policy that is γ /(1−γ) away from the optimal. Let us define the expected total cost for any policy π as J(π) = E s0∼v V π M0 (s 0), measured under some initial state distribution v and the original MDP M 0.Theorem 3.1. There exists an MDP and an imperfect oracleV e (s) with |V e (s) − V * M0,h (s)| =, such that the performance of the induced policy from the cost-to-go oracleπ * = arg min a c(s, a) + γE s ∼Psa [V e (s)] is at least Ω(γ /(1 − γ)) away from the optimal policy π *: DISPLAYFORM0 The proof with the constructed example can be found in Appendix A. DenoteQ DISPLAYFORM1, in high level, we construct an example whereQ e is close to Q * in terms of Q e − Q * ∞, but the order of the actions induced byQ e is different from the order of the actions from Q *, hence forcing the induced policyπ * to make mistakes. As AGGREVATE at best computes a policy that is one-step improvement over the oracle, i.e.,π * = arg min a c(s, a) + γE s ∼Psa [V e (s)], it eventually has to suffer from the above lower bound. This gap in fact is not surprising as AGGREVATE is a one-step greedy algorithm in a sense that it is only optimizing the one-step cost function c from the reshaped MDP M. To see this, note that the cost of the reshaped DISPLAYFORM2, and we havê π * (s) = arg min a E[c (s, a)]. Hence AGGREVATE can be regarded as a special algorithm that aims to optimizing the one-step cost of MDP M that is reshaped from the original MDP M 0 using the cost-to-go oracle. Though when the cost-to-go oracle is imperfect, AGGREVATE will suffer from the above lower bound due to being greedy, when the cost-to-go oracle is perfect, i.e.,V e = V *, being greedy on one-step cost makes perfect sense. To see this, use the property of the cost shaping BID16, we can verify that whenV e = V *: DISPLAYFORM3 Namely the optimal policy on the reshaped MDP M only optimizes the one-step cost, which indicates that the optimal cost-to-go oracle shortens the planning horizon to one: finding the optimal policy on M 0 becomes equivalent to optimizing the immediate cost function on M at every state s. When the cost-to-go oracle is away from the optimality, we lose the one-step greedy property shown in Eq. 4. In the next section, we show that how we can break the lower bound Ω(/(1 − γ)) only with access to an imperfect cost-to-go oracleV e, by being less greedy and looking head for more than one-step. Given the reshaped MDP M withV e as the potential function, as we mentioned in Sec. 2.2, directly optimizing Eq. 2 is as difficult as the original policy search problem, we instead propose to minimize the total cost of a policy π over a finite k ≥ 1 steps at any state s ∈ S: DISPLAYFORM0 Using the definition of cost shaping and telescoping sum trick,we can re-write Eq. 5 in the following format, which we define as k-step disadvantage with respect to the cost-to-go oracle: DISPLAYFORM1 We assume that our policy class Π is rich enough that there always exists a policyπ * ∈ Π that can simultaneously minimizes the k−step disadvantage at every state (e.g., policies in tabular representation in discrete MDPs). Note that when k = 1, minimizing Eq. 6 becomes the problem of finding a policy that minimizes the disadvantage A e M0 (s, a) with respect to the expert and reveals AGGREVATE.The following theorem shows that to outperform expert, we can optimize Eq. 6 with k > 1. Let us denote the policy that minimizes Eq. 6 in every state asπ *, and the value function ofπ * as Vπ *. Theorem 3.2. Assumeπ * minimizes Eq. 6 for every state s ∈ S with k > 1 and |V e (s) − V * (s)| = Θ, ∀s. We have: DISPLAYFORM2 Compare the above theorem to the lower bound shown in Theorem 3.1, we can see that when k > 1, we are able to learn a policy that performs better than the policy induced by the oracle (i.e.,π * (s) = arg min aQ e (s, a)) by at least (DISPLAYFORM3 The proof can be found in Appendix B. Theorem 3.2 and Theorem 3.1 together summarize that when the expert is imperfect, simply computing a policy that minimizes the one-step disadvantage (i.e., (k = 1)) is not sufficient to guarantee near-optimal performance; however, optimizing a k-step disadvantage with k > 1 leads to a policy that guarantees to outperform the policy induced by the oracle (i.e., the best possible policy that can be learnt using AGGREVATE and AGGREVATED). Also our theorem provides a concrete performance gap between the policy that optimizes Eq. 6 for k > 1 and the policy that induced by the oracle, which is missing in previous work (e.g., BID5).As we already showed, if we set k = 1, then optimizing Eq. 6 becomes optimizing the disadvantage over the expert A e M0, which is exactly what AGGREVATE aims for. When we set k = ∞, optimizing Eq. 6 or Eq. 5 just becomes optimizing the total cost of the original MDP. Optimizing over a shorter Reset system. Execute π θn to generate a set of trajectories {τ i} N i=1. Reshape cost c (s t, a t) = c(s t, a t) + V e t+1 (s t+1) − V e t (s t), for every t ∈ [1, |τ i |] in every trajectory τ i, i ∈ [N]. Compute gradient: DISPLAYFORM0 8:Update disadvantage estimator toÂ πn,k M using {τ i} i with reshaped cost c. Update policy parameter to θ n+1. 10: end for finite horizon is easier than optimizing over the entire infinite long horizon due to advantages such as smaller variance of the empirical estimation of the objective function, less temporal correlations between states and costs along a shorter trajectory. Hence our main theorem essentially provides a tradeoff between the optimality of the solutionπ * and the difficulty of the underlying optimization problem. Given the original MDP M 0 and the cost-to-go oracleV e, the reshaped MDP's cost function c is obtained from Eq. 1 using the cost-to-go oracle as a potential function. Instead of directly applying RL algorithms on M 0, we use the fact that the cost-to-go oracle shortens the effective planning horizon of M, and propose THOR: Truncated HORizon Policy Search summarized in Alg. 1. The general idea of THOR is that instead of searching for policies that optimize the total cost over the entire infinitely long horizon, we focus on searching for polices that minimizes the total cost over a truncated horizon, i.e., a k−step time window. Below we first show how we derive THOR from the insight we obtained in Sec. 3.Let us define a k-step truncated value function V π,k M and similar state action value function Q π,k M on the reshaped MDP M as: DISPLAYFORM0 At any time state s, V π,k M only considers (reshaped) cost signals c from a k-step time window. We are interested in searching for a policy that can optimizes the total cost over a finite k-step horizon as shown in Eq. 5. For MDPs with large or continuous state spaces, we cannot afford to enumerate all states s ∈ S to find a policy that minimizes the k−step disadvantage function as in Eq. 5. Instead one can leverage the approximate policy iteration idea and minimize the weighted cost over state space using a state distribution ν BID11 BID2: DISPLAYFORM1 For parameterized policy π (e.g., neural network policies), we can implement the minimization in Eq. 10 using gradient-based update procedures (e.g., Stochastic Gradient Descent, Natural Gradient BID10 BID1) in the policy's parameter space. In the setting where the system cannot be reset to any state, a typical choice of exploration policy is the currently learned policy (possibly mixed with a random process BID12 to futher encourage exploration). Denote π n as the currently learned policy after iteration n and P r πn (·) as the average state distribution induced by executing π n (parameterized by θ n) on the MDP. Replacing the exploration distribution by P r πn (·) in Eq. 10, and taking the derivative with respect to the policy parameter θ, the policy gradient is: DISPLAYFORM2 where τ k ∼ π n denotes a partial k−step trajectory τ k = {s 1, a 1, ..., s k, a k |s 1 = s} sampled from executing π n on the MDP from state s. Replacing the expectation by empirical samples from π n, replacing Q π,k M by a critic approximated by Generalized disadvantage Estimator (GAE)Â π,k M, we get back to the gradient used in Alg. 1: DISPLAYFORM3 where |τ | denotes the length of the trajectory τ. If using the classic policy gradient formulation on the reshaped MDP M we should have the following expression, which is just a re-formulation of the classic policy gradient BID29: DISPLAYFORM0 which is true since the cost c i (we denote c i (s, a) as c i for notation simplicity) at time step i is correlated with the actions at time step t = i all the way back to the beginning t = 1. In other words, in the policy gradient format, the effectiveness of the cost c t is back-propagated through time all the way back the first step. Our proposed gradient formulation in Alg. 1 shares a similar spirit of Truncated Back-Propagation Through Time BID30, and can be regarded as a truncated version of the classic policy gradient formulation: at any time step t, the cost c is back-propagated through time at most k-steps: DISPLAYFORM1 In Eq. 13, for any time step t, we ignore the correlation between c t and the actions that are executed k-step before t, hence elimiates long temporal correlations between costs and old actions. In fact, AGGREVATE D BID25, a policy gradient version of AGGREVATE, sets k = 1 and can be regarded as No Back-Propagation Through Time. The above gradient formulation provides a natural half-way point between IL and RL. When k = 1 andV e = V * M0 (the optimal value function in the original MDP M 0): DISPLAYFORM0 where, for notation simplicity, we here use E τ to represent the expectation over trajectories sampled from executing policy π θ, and A π * M0 is the advantage function on the original MDP M 0. The fourth expression in the above equation is exactly the gradient proposed by AGGREVATED BID25. AGGREVATED performs gradient descent with gradient in the format of the fourth expression in Eq. 14 to discourage the log-likelihood of an action a t that has low advantage over π * at a given state s t.On the other hand, when we set k = ∞, i.e., no truncation on horizon, then we return back to the classic policy gradient on the MDP M obtained from cost shaping withV e. As optimizing M is the same as optimizing the original MDP M 0 , our formulation is equivalent to a pure RL approach on M 0. In the extreme case when the oracleV e has nothing to do with the true optimal oracle V *, as there is no useful information we can distill from the oracle and RL becomes the only approach to solve M 0. We evaluated THOR on robotics simulators from OpenAI Gym BID4. Throughout this section, we report reward instead of cost, since OpenAI Gym by default uses reward. The baseline we compare against is TRPO-GAE and AGGREVATED BID25.To simulate oracles, we first train TRPO-GAE until convergence to obtain a policy as an expert π e. We then collected a batch of trajectories by executing π e. Finally, we use TD learning to train a value functionV e that approximates V e. In all our experiments, we ignored π e and only used the pre-trainedV e for reward shaping. Hence our experimental setting simulates the situation where we only have a batch of expert demonstrations available, and not the experts themselves. This is a much harder setting than the interactive setting considered in previous work BID20 BID25 BID5. Note that π e is not guaranteed to be an optimal policy, and V e is only trained on the demonstrations from π e, therefore the oracleV e is just a coarse estimator of V * M0. Our goal is to show that, compared to AGGREVATED, THOR with k > 1 in significantly better performance; compared to TRPO-GAE, THOR with some k << H converges faster and is more sample efficient. For fair comparison to RL approaches, we do not pre-train policy or criticÂ using demonstration data, though initialization using demonstration data is suggested in theory and has been used in practice to boost the performance BID20 BID3.For all methods we report statistics (mean and standard deviation) from 25 seeds that are i.i.d generated. For trust region optimization on the actor π θ and GAE on the critic, we simply use the recommended parameters in the code-base from TRPO-GAE. We did not tune any parameters except the truncation length k. We consider two discrete action control tasks with sparse rewards: Mountain-Car, Acrobot and a modified sparse reward version of CartPole. All simulations have sparse reward in the sense that no reward signal is given until the policy succeeds (e.g., Acrobot swings up). In these settings, pure RL approaches that rely on random exploration strategies, suffer from the reward sparsity. On the other Note that in our setting whereV e is imperfect, THOR with k > 1 works much better than AG-GREVATED (THOR with k = 1) in Acrobot. In Mountain Car, we observe that AGGREVATED achieves good performance in terms of the mean, but THOR with k > 1 (especially k = 10) in much higher mean+std, which means that once THOR receives the reward signal, it can leverage this signal to perform better than the oracles. We also show that THOR with k > 1 (but much smaller than H) can perform better than TRPO-GAE. In general, as k increases, we get better performance. We make the acrobot setting even harder by setting H = 200 to even reduce the chance of a random policy to receive reward signals. FIG1 to FIG1, we can see that THOR with different settings of k always learns faster than TRPO-GAE, and THOR with k = 50 and k = 100 significantly outperform TRPO-GAE in both mean and mean+std. This indicates that THOR can leverage both reward signals (to perform better than AGGREVATED) and the oracles (to learn faster or even outperform TRPO). We tested our approach on simulators with continuous state and actions from MuJoCo simulators: a modified sparse reward Inverted Pendulum, a modifed sparse reward Inverted Double Pendulum, Hopper and Swimmer. Note that, compared to the sparse reward setting, Hopper and Swimmer do not have reward sparsity and policy gradient methods have shown great BID21 BID23. Also, due to the much larger and more complex state space and control space compared to the simulations we consider in the previous section, the value function estimatorV e is much less accurate in terms of estimating V * M0 since the trajectories demonstrated from experts may only cover a very small part of the state and control space. FIG2 shows the of our approach. For all simulations, we require k to be around 20% ∼ 30% of the original planning horizon H to achieve good performance. AGGREVATED (k = 1) learned very little due to the imperfect value function estimatorV e. We also tested k = H, where we observe that reward shaping withV e gives better performance than TRPO-GAE. This empirical observation is consistent with the observation from BID16 BID16 ) used SARSA (, not policy gradient based methods). This indicates that even whenV e is not close to V *, policy gradient methods can still employ the oracleV e just via reward shaping. Finally, we also observed that our approach significantly reduces the variance of the performance of the learned polices (e.g., Swimmer in FIG2) in all experiments, including the sparse reward setting. This is because truncation can significantly reduce the variance from the policy gradient estimation when k is small compared to H. We propose a novel way of combining IL and RL through the idea of cost shaping with an expert oracle. Our theory indicates that cost shaping with the oracle shortens the learner's planning horizon as a function of the accuracy of the oracle compared to the optimal policy's value function. Specifically, when the oracle is the optimal value function, we show that by setting k = 1 reveals previous imitation learning algorithm AGGREVATED. On the other hand, we show that when the oracle is imperfect, using planning horizon k > 1 can learn a policy that outperforms a policy that would been learned by AGGREVATE and AGGREVATED (i.e., k = 1). With this insight, we propose THOR (Truncated HORizon policy search), a gradient based policy search algorithm that explicitly focusing on minimizing the total cost over a finite planning horizon. Our formulation provides a natural half-way point between IL and RL, and experimentally we demonstrate that with a reasonably accurate oracle, our approach can outperform RL and IL baselines. We believe our high-level idea of shaping the cost with the oracle and then focusing on optimizing a shorter planning horizon is not limited to the practical algorithm we proposed in this work. In fact our idea can be combined with other RL techniques such as Deep Deterministic Policy Gradient (DDPG) BID12, which has an extra potential advantage of storing extra information from the expert such as the offline demonstrations in its replay buffer BID28 ). Though in our experiments, we simply used some expert's demonstrations to pre-trainV e using TD learning, there are other possible ways to learn a more accurateV e. For instance, if an expert is available during training BID20, one can online updateV e by querying expert's feedback. A PROOF OF THEOREM 3.1 Figure 3: The special MDP we constructed for theorem 3.1Proof. We prove the theorem by constructing a special MDP shown in Fig 3, where H = ∞. The MDP has deterministic transition, 2H + 2 states, and each state has two actions a 1 and a 2 as shown in Fig. 3. Every episode starts at state s 0. For state s i (states on the top line), we have c(s i) = 0 and for state s i (states at the bottom line) we have c(s i) = 1.It is clear that for any state s i, we have Q * (s i, a 1) = 0, Q * (s i, a 2) = γ, Q * (s i, a 1) = 1 and Q * (s i, a 2) = 1 + γ, for i ≥ 1. Let us assume that we have an oracleV e such thatV e (s i) = 0.5 + δ and V e (s i) = 0.5 − δ, for some positive real number δ. Hence we can see that |V e (s) − V * (s)| = 0.5 + δ, for all s. DenoteQ e (s, a) = c(s, a) + γE s ∼Psa [V e (s)], we know thatQ e (s i, a 1) = γ(0.5 + δ),Q e (s i, a 2) = γ(0.5 − δ),Q e (s i, a 1) = 1 + γ(0.5 + δ) andQ e (s i, a 2) = 1 + γ(0.5 − δ).It is clear that the optimal policy π * has cost J(π *) = 0. Now let us compute the cost of the induced policy from oracleQ e:π(s) = arg min aQ e (s, a). As we can seeπ makes a mistake at every state as arg min aQ e (s, a) = arg min a Q * (s, a). Hence we have J(π) = γ 1−γ. Recall that in our constructed example, we have = 0.5 + δ. Now let δ → 0 + (by δ → 0 + we mean δ approaches to zero from the right side), we have → 0.5, hence J(π) = Proof of Theorem 3.2. In this proof, for notation simplicity, we denote V π M0 as V π for any π. Using the definition of value function V π, for any state s 1 ∈ S we have: DISPLAYFORM0

Represent the TLDR for finding the corresponding scientific article

Combining Imitation Learning and Reinforcement Learning to learn to outperform the expert

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question: Represent the article for finding a single-sentence TLDR

Question: Recently, Generative Adversarial Networks (GANs) have emerged as a popular alternative for modeling complex high dimensional distributions. Most of the existing works implicitly assume that the clean samples from the target distribution are easily available. However, in many applications, this assumption is violated. In this paper, we consider the observation setting in which the samples from a target distribution are given by the superposition of two structured components, and leverage GANs for learning of the structure of the components. We propose a novel framework, demixing-GAN, which learns the distribution of two components at the same time. Through extensive numerical experiments, we demonstrate that the proposed framework can generate clean samples from unknown distributions, which further can be used in demixing of the unseen test images. In this paper, we consider the classical problem of separating two structured signals observed under the following superposition model: DISPLAYFORM0 where X ∈ X and N ∈ N are the constituent signals/components, and X, N ⊆ R p denote the two structured sets. In general the separation problem is inherently ill-posed; however, with enough structural assumption on X and N, it has been established that separation is possible. Depending on the application one might be interested in estimating only X (in this case, N is considered as the corruption), which is referred to as denoising, or in recovering (estimating) both X and N which is referred to as demixing. Both denoising and demixing arise in a variety of important practical applications in the areas of signal/image processing, computer vision, machine learning, and statistics BID7 BID11 BID3 BID6. Most of the existing demixing techniques assume some prior knowledge on the structures of X and N in order to recover the desired component signals. Prior knowledge about the structure of X and N can only be obtained if one has access to the generative mechanism of the signals and clean samples from the probability distribution defined over sets X and N. In many practical settings, none of these may be feasible. In this paper, we consider the problem of separating constituent signals from superposed observations when we do not have access to the clean samples from any of the distributions (fully unsupervised approach). In particular, we are given a set of superposed observations DISPLAYFORM1 where X i ∈ X and Y i ∈ N are i.i.d samples from their respective (unknowns) distributions. In this setup, we explore two questions: First, How can one learn the prior knowledge about the individual components from superposition samples?; hence, we concern with a learning problem. Second, Can we leverage the implicitly learned constituent distributions for tasks such as demixing of a test image?. As a , in the latter question, we deal with a inference task. Motivated by the recent success of generative models in high-dimensional statistical inference tasks such as compressed sensing in BID4, in this paper, we focus on Generative Adversarial Network (GAN) based generative models to implicitly learn an unknown distribution, i.e., generate samples from the learned distribution. Most of the existing works on GANs typically assume 2 PRIOR ART To overcome the inherent ambiguity issue in problem, many existing methods have assumed that the structure of sets X and N (i.e., the structures can be low-rank matrices, or have sparse representation in some domain BID23) is a prior known and also that the signals from X and N are "distinguishable" BID10 BID27 BID9 BID12. Knowledge of the structures is a big restriction in many real-world applications. Recently, there have been some attempts to automate this hardcoding approach. Among them, structured sparsity BID17, dictionary learning BID10, and in general manifold learning are the prominent ones. While these approaches have been successful to some extent, they still cannot fully address the need for the prior structure. Over the last decade, deep neural networks have been demonstrated that they can learn useful representations of real-world signals such as natural images, and thus have helped us to understand the structure of the high dimensional signals, for e.g., using deep generative models BID30. In this paper, we focus on Generative Adversarial Networks GANs) BID14 as the generative models for implicitly learning the distribution of constituent components. GANs have been established as a very successful tool for generating structured high-dimensional signals BID2 BID31 as they do not directly learn a probability distribution; instead, they generate samples from the target distribution(s) BID13. In particular, if we assume that the structured signals are drawn from a distribution lying on a low-dimensional manifold, GANs generate points in the high-dimensional space that resemble those coming from the true underlying distribution. Since their inception, there has been a flurry of works on GANs BID35 BID33 BID29 to name a few. In most of the existing works on GANs with a few notable exceptions such as BID32; BID5; BID20 BID16; BID34, it is implicitly assumed that one has access to clean samples of the desired signal. However, in many practical scenarios, the desired signal is often accompanied by unnecessary components. Recently, GANs have also been used for capturing the structure of high-dimensional signals specifically for solving inverse problems such as sparse recovery, compressive sensing, and phase retrieval BID4 BID20 BID16. For instance, BID4 has shown that generative models provide a good prior for structured signals, for e.g., natural images, under compressive sensing settings over sparsity-based recovery methods. They rigorously analyze the statistical properties of a generative model based on compressed sensing and provide some theoretical guarantees and experimental evidence to support their claims. However, they don't explicitly propose an optimization procedure to solve the recovery problem. They simply suggest using stochastic gradient-based methods in the low-dimensional latent space to recover the signal of interest. This has been addressed in BID26, where the authors propose using a projected gradient descent algorithm for solving the recovery problem directly in the ambient space (space of the desired signal). They provide theoretical guarantees for the convergence of their algorithm and also demonstrate improved empirical over BID4. While GANs have found many applications, most of them need direct access to the clean samples from the unknown distribution, which is not the case in many real applications such as medical imaging. AmbientGAN framework BID5 partially addresses this problem. In particular, they studied various measurement models and showed that their GAN model can find samples of clean signals from corrupted observations. However, AmbientGAN assumes that the observation model and parameters are known. That is, they have assumed that one has access to the samples of corruption part, which is a strong restriction in the real-world applications. One of our main contributions is addressing this limitation by studying the demixing problem. Actually, if we can learn the distribution of both components (e.g., generating samples from them), then we can use the samples of the second component (corruption part) for downstream task such as denoising without explicitly needing the samples from the corruption part. That is why our framework is a purely unsupervised approach. In addition, AmbientGAN just learns the distribution of the clean images; however, it has not been used for the task of image denoisng (i.e., how to denoise an unseen corrupted image). Our framework addresses this issue in a general scenario of demixing of unseen test images.3 AND PROPOSED IDEA 3.1 PRELIMINARIES Generative Adversarial Networks (GANs) are one of the successful generative models in practice was first introduced by BID14 for generating samples from an unknown target distribution. As opposed to the other approaches for density estimation such as Variational AutoEncoders (VAEs) BID21, which try to learn the distribution itself, GANs are designed to generate samples from the target probability density function. This is done through a zero-sum game between two players, generator, G and discriminator, D in which the generator G plays the role of producing the fake samples and discriminator D plays the role of a cop to find the fake and genuine samples. Mathematically, this is accomplished through the following min-max optimization problem: DISPLAYFORM0 where θ g and θ d are the parameters of generator network and discriminator network, respectively. D x denotes the target probability distribution, and D z represents the probability distribution of the hidden variables z ∈ R h, which is assumed either a uniform distribution in [−1, 1] h, or standard normal. One can also use identity function instead of log function in the above expression. The ing formulation is called WGAN A. et al.. It has been shown that if G and D have enough capacity, then solving optimization problem by alternative stochastic gradient descent guarantees the distribution D g at the output of the generator converges to D x. Having discussed the basic setup of GANs, next we present the proposed modifications to the basic GAN setup that allows for usage of GANs as a generative model for demixing structured signals. In this section, we discuss our main contribution. First, we start with learning problem (the first question in the introduction section) and explain our GAN framework for learning the distribution of two constituent components. Next, we move to the inference part (the second question in the introduction section) to show that how we can leverage the learned generating process for demixing of a test mixed image. Finally, we provide some theoretical intuition about the success/failure of the demixing-GAN. FIG0 shows the GAN architecture, we use for the purpose of separating or demixing of two structured signals form their superposition. As illustrated, we have used two generators and have fed them with two random noise vectors z 1 ∈ R h1 and z 2 ∈ R h2 according to a uniform distribution defined on a hyper-cube, where h 1, h 2 are less than the dimension of the input images. We also assume that they are independent of each other. Next, the output of generators are summed up and the is fed to the discriminator along with the superposition samples, y i s. In FIG0, we just show the output of each generator after training for an experiment case in which the mixed images consist of 64 MNIST binary image (for X part) and a second component constructed by random sinusoidal (for N part) (please see the experiment section for more details). Somewhat surprisingly, the architecture based on two generators can produce samples from the distribution of each component after enough number of training iterations. We note that this approach is fully unsupervised as we only have access to the mixed samples and nothing from the samples of constituent components is known. As mentioned above, this is in sharp contrast with the AmbientGAN. As a , the demixing-GAN framework can generate samples from both components (If for example the second component is drawn from a random sinusoidal, then the generated samples can be used in the task of denoising where the corruption components are sampled from highly structured sinusoidal waves). Now, we can use the trained generators in FIG0 for demixing of the constituent components for a given test mixed image which has not been used in training. To this end, we use our assumption that the components have some structure and the representation of this structure is given by the last layer of the trained generator. This observation together with this fact that in GANs, the low-dimension random vector z is representing the hidden variables, leads us to this point: in order to demix a new test mixed image, we have to find a hidden representations corresponding to each component which give the smallest distance to the constituent images in the space of G θg 1 and G θg 2 BID26 BID4. In other words, we have to solve the following optimization problem: DISPLAYFORM0 where u denotes the test mixed image. Now, each component can be estimated by evaluating DISPLAYFORM1 and G θg 2 (z 2) 1. While the optimization problem in is non-convex, we can still solve it through an alternative minimization fashion. We note that in optimization problem, we did not project on the box sets on which z 1 and z 2 lie on. Instead we have used regularizer terms in the objective functions (which are not meant as projection step). We empirically have observed that imposing these regularizers can help to obtain good quality images in our experiment; plus, they may help that the gradient flow to be close in the region of interest by generators. This is also used in BID4. Now, we provided some theoretical intuitions for the demixing-GAN. Recall that the superposition model is given by Y = X + N, and D y, D x and D n denote the distribution of Y, X, and N, respectively. Let DISPLAYFORM0 denotes the joint distribution of the hidden random vectors 1 G θg 1 and G θg 2 denote the first and second trained generator with parameter θg 1 and θg 2, respectively.with marginal probability as D zi for i = 1, 2. We note that in demxing setting there are not samples from the component N as opposed to the typical denoising scenarios. Now we have the following mini-max loss as: DISPLAYFORM1 Following the standard GAN framework, for the fixed G 1 and G 2, we have: DISPLAYFORM2 where DISPLAYFORM3. Hence, the optimal discriminator is given by D * = Dx * Dn Dx * Dn+Dg 1 * Dg 2 since Y = X + N and the fact that D y and D g are the pdf and defined over. This means that the global optimal of the above optimization problem is achieved iff D x * D n = D g1 * D g2 (* denotes the convolution operator). However, this condition is generally an ill-posed equation. That is, in general, D x = D g1 and D n = D g2. In the best case, we can have hope to uniquely determine the distributions up to a permutation (similar thing is also true for the ICA method). This is the point actually we need some notion of incoherence between two constituent structures, X, and N. So, the question is this under what incoherent condition, we can have a well-conditioned equation? Also, by using the above optimality condition and taking the Fourier transform (or using the characteristic function, Φ) from this optimality equation, we obtain: DISPLAYFORM4 ). For DISPLAYFORM5 ) to be well-defined, a straightforward condition is that the Φ n should be non-zero almost everywhere. As a , even if we somehow figure out the right incoherent condition, D x is uniquely determined by D n if the Fourier transform of D n is non-zero. While we currently do not have a right answer for the above question, we conjecture that in addition to the incoherence issue in the signal domain, the hidden space (z-space) in both generators play an important role to make the demixing problem possible. We investigate this idea and the other things empirically in the experiment section. In this section, we present various experiments showing the efficacy of the proposed framework (depicted in FIG0) in two different setups. First, we will focus on learning the structured distributions from the superposition observation samples. Next, we explore the use of generative models from the proposed GAN framework in an inference task. In all the following experiments, we did our best for choosing all the hyper-parameters. In this section, we present the of our experiments for learning the distribution of the constituent components on different datasets. We first present our experiment with MNIST dataset, and then we show the similar set of experiments with Fashion-MNIST dataset (F-MNIST) BID15. Next, we illustrate the performance of the demixing-GAN on Quick-Draw Qui (a). Finally, we present some experimental investigation for the conditions under which the demixing-GAN is failed. We start the experiments with considering four sets of constituent components. We have used the network architectures for discriminator and generators similar to the one proposed in DCGAN BID25. DCGAN is a CNN based-GAN consists of convolutional layers followed by batch normalization (except the last layer of the generator and first layer of discriminator). In the first two experiments in this section, we use the superposition of the MNIST digits and some random corruption signals. That is, we mix the MNIST digits with two (structured) corruption signals: random sinusoidal waves, and random vertical and horizontal lines. In particular, first we generate random sinusoidal waves in which the amplitude, frequency, and phase are random numbers, and second we construct the random vertical and horizontal lines. In FIG1, we show the training evolution of two fixed random vectors, z 1 and z 2 in R 100 in the output of two generators. In the top left panel, we have added one random sinusoidal wave to the clean digits. As we can see, our proposed GAN architecture can learn two distributions and generate samples from each of them. In the bottom left panel, we repeat the same experiment with random vertical and horizontal lines as the second component (two random vertical and two random horizontal lines are added to the clean digits). While there is some notion of mode collapse, still two generators can produce the samples from the distribution of the constituent components. For the third experiment in this section, our mixed images comprise of two MNIST digits from 0 to 9. In this case, we are interested in learning the distribution from which each of the digits is drawn. Similar to the previous cases, the top right panel in FIG1 shows the evolution of two fixed random vectors, z 1 and z 2. As we can see, after 32 epochs, the output of the generators would be the samples of MNIST digits. Finally, in the last experiment of this section, we generate the mixed images as the superposition of digits 1 and 2. In the training set of MNIST dataset, there are around 6000 samples from each digit of 1 and 2. We have used these digits to form the set of superposition images. The bottom right panel of FIG1 shows the output of two generators, which can learn the distribution of two digits. The interesting point in these experiments is that each GAN can learn the different variety of existing digits in MNIST training dataset, and we typically do not see mode collapse, which is a major problem in the training of GANs BID13. In this section, we illustrate the performance of the proposed demixing-GAN for F-MNIST dataset. The training dataset in F-MNIST includes 60000 gray-scale images with size of 28 × 28 classified in 10 classes. The different labels denote objects, which include T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, and Ankle boot. Similar to the experiments with MNIST dataset being illustrated in FIG1, we train the demixing-GAN where we have used InfoGAN architecture for the generators. The architecture of the generators in InfoGAN is very similar to the DCGAN discussed above with the same initialization procedure. The dimension of input noise to the generators is set to 62. We have also used the same discriminator in DCGAN. Left panel in Figure 3 shows the output of two generators, which can learn the distribution of dress and bag images during 21 epochs from mixed dress and bag images. Figure 3: Left Panel: evolution of output samples by two generators for fixed z1 and z2. The mixed images comprise only two objects, dress, and bag in the training F-MNIST dataset. One generator produces the samples from dress distribution, while the other one outputs the samples from the bag distribution. Right panel: the performance of the trained generators for demixing of two constituent components (inference). The first two columns are the ground-truth components. The third column is the ground-truth mixed image and the last two columns denote the recovered components. The first row uses the same generator trained for only one digit (drawn from MNIST test dataset) and a random sinusoidal. The second row uses the generator trained only for digits 1 and 2. The last row shows the of demixing with ICA method. We now test the performance of two trained generators in a demixing scenario for the test mixed images, which have not been seen in the training time. Right panel in Figure 3 shows our experiment in which we have illustrated the demixing on three different test mixed images. Here, we have compared the performance of the demixing-GAN with Independent component analysis (ICA) method BID18 ) (We have implemented ICA using Scikit-learn module BID24). In the top and middle rows of Figure 3 (right panel), we consider the mixed images generated by adding a digit (drawn from MNIST test dataset) and a random sinusoidal. Then the goal is to separate (demix) these two from their given sum. To do this, we use the GAN trained for learning the distribution of digits and sinusoidal waves (the top left panel of FIG1) and solve the optimization problem in through an alternative minimization approach. As a , we obtain z 1 and z 2. The corresponding constituent components is then obtained by evaluating G θg 1 (z 1) and G θg 2 (z 2). In the right panel of Figure 3, the first two columns denote the ground-truth of the constituent components. The middle one is the mixed ground-truth, and the last two show the recovered components using demixing-GAN and ICA. In the last row, digits 1 and 2 drawn from the MNIST test dataset, are added to each other and we apply the GAN trained for learning the distribution of digits 1 and 2 (bottom right panel in FIG1). As we can see, our proposed GAN can separate two digits; however, ICA method fails in demixing of two components. In addition, Table 1 has compared numerically the quality of recovered components with the corresponding ground-truth ones through mean square error (MSE) and Peak Signal-to-Noise Ratio (PSNR) criteria. Now, we evaluate the performance of the trained demixing-GAN on the F-MNIST dataset. For each panel in Figure 4, the first two columns denote two objects from F-MNIST test dataset as the ground-truth components. The third column is the ground-truth mixed image, and the last two columns show the recovered constituent components similar to the previous case. The top left uses the generator trained for only two objects for 20 epochs. The top right uses the generator trained for all 10 objects for 20 epochs. The bottom left uses the same generator trained for only two objects for 30 epochs. The bottom right shows the of demixing with ICA method. As we can see, ICA fails to separate the components (images of F-MNIST) from each other, while the proposed demixing-GAN can separate the mixed images from each other. While the estimated image components are not exactly matched to the ground-truth ones (first two columns), they are semantically similar to the ground-truth components. In this section, we explore the performance of the demixing-GAN when the superposed images is the sum of digits 8 from MNIST dataset and dresses from the F-MNIST dataset. The experiment for this setup has been illustrated in the left panel of FIG2. Since our goal is to separate dress from the digit 8, for the first generator, we have used the InfoGAN architecture being used in the experiment demixing-GAN demixing-GAN demixing-GAN ICA Method Figure 4: Performance of the trained generators for demixing of two constituent components. In all panels, the first two columns are the ground-truth components. The third column is the ground-truth mixed image and the last two columns denote the recovered components. The top left uses the generator trained for only two objects for 20 epochs. The top right uses the generator trained for all 10 objects for 20 epochs. The bottom left uses the same generator trained for only two objects for 30 epochs. The bottom right shows the of demixing with ICA method.in section 4.1.2 and similarly the DCGAN architecture for the second generator as section 4.1.1. As a , the input noise to the first generator is drawn uniformly from [−1, 1] 62, and uniformly from [−1, 1] 100 for the second generator. The left panel of FIG2 shows the evolution of output samples by two generators for fixed z 1 and z 2. As we can see, after 21 epoch, the first generator is able to generate dress samples and the second one outputs samples of digit 8. Similar to the previous testing scenarios, we now evaluate the performance of the demixing-GAN in comparison with ICA for separating a test image which is a superposition of a digit 8 drawn randomly from MNIST test dataset and dress object drawn randomly from F-MNIST test dataset. Right panel in FIG2 shows the performance of the demixng-GAN and ICA method. As we can see, ICA fails to demix two images from their superposition, whereas the demixing-GAN is able to separate digit 8 very well and to some extend the dress object from the input superposed image. MSE and PSNR values for the first component using ICA recovery method is given by 0.40364 and 3.94005, respectively. Also, MSE and PSNR for the first component using ICA recovery method is given by 0.15866 and 7.99536, respectively. In this section, we present our demixing framework for another dataset, Quick-Draw dataset (Qui, a) released by Google recently. The Quick Draw Dataset is a collection of 50 million drawings categorized in 345 classes, contributed by players of the game Quick, Draw! Qui (b). For the the experiment in the left panel of FIG3, we consider only two objects, face, and flower in the Quick Draw Dataset (training set includes 16000 images of size 28 × 28 for each class). As a , the input mixed images are the superposition of different faces and flowers. The left panel of FIG3 shows the evolution of the random vectors z 1 and z 2 (drawn uniformly from [−1, 1] 64 ). As we can see, after 31 epochs, one generator can produce various kind of faces, while the other one generates different shapes of flowers. Now, we consider a more challenging scenario in which the constituent components in the mixed images are just airplane shapes. That is, we randomly select the airplane shapes from 16000 images in the training set, and add them together to construct the input mixed images. We have been noticed that in the 16000 images of the airplane shapes, in general, there are two structures. One is related to the airplanes having been drawn by the players in more simple and somehow flat manner (they are mostly similar to an ellipse with or without wings) in the Quick, Draw game, while the second one consists of the more detailed shapes (they have a tail and maybe with different orientation). Right panel of FIG3 depicts the performance of the demixing-GAN for this setup. One surprising point is that while both components in the superposition are drawn from one class (e.g., airplane shapes), the demixing-GAN is still able to demix the hidden structure in the airplane distribution. Thus, we think that just having the same distribution for both of the constituent components is not necessarily a barrier for demixing performance. We conjecture that somehow different features of the shapes drawn from the same distribution makes demixing possible by forcing the enouth incoherence between the components. As we can see, after 31 epochs, both generators can learn two mentioned structures, and regarding two structures, they can cluster the shape of airplanes into two types. In this section, we empirically explore our observation about the failure of the demixing-GAN. We focus on two spaces, hidden space (z-space) and signal or generator space (the output of generators) in discovering the failure of demixing-GAN.Our first observation concerns the z-space. We observe that if the hidden vectors form z-space of two generators are aligned to each other, then the two generators cannot output the samples in the signal space, representing the distribution of the constituent components. To be more precise, in the left panel of Figure 7, we consider separating digits 8 and 2 from their superpositions. However, here, we feed both generators with the same vector, i.e., z 1 = z 2 in each batch (this is considered as the extreme case where precisely the hidden variables equal to each other) and track the evolution of the output samples generated by both generators. As we can see, even after 21 epochs, the generated samples by both generators are an unclear combination of both digits 2 and 8, and they are not separated clearly as opposed to the case when we feed the generators with i.i.d random vectors. We also repeat the same experiment with two aligned vectors z 1 and z 2, i.e., z 2 = 0.1z 1, the right panel of Figure 7 shows the evolution of the output samples generated by both generators for this setup. As shown in this experiment, two generators cannot learn the distribution of digits 8 and 2. While we do not currently have a mathematical argument for this observation, we conjecture that the hidden space (z-space) is one of the essential pieces in the demixing performance of the proposed demixing-GAN. We think that having (random) independent or close orthogonal vector z's for the input of each generator is a necessary condition for the success of learning the distribution of the constituent components, and consequently demixing of them. Further investigation of this line of study is indeed an interesting research direction, and we defer it for future research. In addition to the hidden space, here we design some experiments in the generator space that reveals the condition under which the demixing is failed. In particular, we consider the airplane images in Quick-Draw dataset. To construct the input mixed images, we consider randomly chosen images of the airplane from 16000 images as the first component. Then, the second component is constructed by rotating exactly the same one in the first components in a counterclockwise direction. We consider 4 different rotations, 0 DISPLAYFORM0 •. This experiment is sort of similar to the one in the right panel of FIG3 in which we have seen that demixing-GAN can capture the internal structure in the airplane dataset by clustering them into two types. Now we perform the demixing-GAN on these datasets. FIG4 illustrated the the evolution of the generators for various rotation degrees. The top left panel shows the case in which exactly both components are the same. Obviously, the demixing, in this case, is impossible as there is no hope to distinguish the components from each other. Moving on, in the other panels of FIG4, we have different rotation settings. As we can see, once we move forward to the 90•, both generators can capture the samples from the airplane distribution; however, as not clear as the case in which we had added the airplane shapes randomly to construct the input mixed images. We conjecture that changing the orientation of one component can make it incoherent to some extent from the other component, and consequently makes the demixing possible. In other words, we see again when two images show some distinguishable structures (in this case, the first one has 0-oriented object and the other is the same object, but rotated 90• counterclockwise), then the demixing-GAN can capture these structures. In this paper, we considered a GAN framework for learning the structure of the constituent components in a superposition observation model. We empirically showed that it is possible to implicitly learn the underlying distribution of each component and use them in the downstream task such as demixing of a test mixed image. We also investigated the conditions under which the proposed demixing framework fails through extensive experimental simulations and provided some theoretical insights.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

An unsupervised learning approach for separating two structured signals from their superposition

Represent the TLDR for finding the corresponding scientific article.

Parametric adversarial divergences implicitly define more meaningful task losses for generative modeling, we make parallels with structured prediction to study the properties of these divergences and their ability to encode the task of interest.