Question: Represent the text to find a very short summary (1-2 sentences max).

Question: We present an adversarial exploration strategy, a simple yet effective imitation learning scheme that incentivizes exploration of an environment without any extrinsic reward or human demonstration. Our framework consists of a deep reinforcement learning (DRL) agent and an inverse dynamics model contesting with each other. The former collects training samples for the latter, and its objective is to maximize the error of the latter. The latter is trained with samples collected by the former, and generates rewards for the former when it fails to predict the actual action taken by the former. In such a competitive setting, the DRL agent learns to generate samples that the inverse dynamics model fails to predict correctly, and the inverse dynamics model learns to adapt to the challenging samples. We further propose a reward structure that ensures the DRL agent collects only moderately hard samples and not overly hard ones that prevent the inverse model from imitating effectively. We evaluate the effectiveness of our method on several OpenAI gym robotic arm and hand manipulation tasks against a number of baseline models. Experimental show that our method is comparable to that directly trained with expert demonstrations, and superior to the other baselines even without any human priors. Over the past decade, imitation learning (IL) has been successfully applied to a wide range of domains, including robot learning BID7 BID20, autonomous navigation BID4 BID19, manipulation tasks BID11 BID17, and self-driving cars BID5. Traditionally, IL aims to train an imitator to learn a control policy π only from expert demonstrations. The imitator is typically presented with multiple demonstrations during the training phase, with an aim to distill them into π. To learn π effectively and efficiently, a large set of high-quality demonstrations are necessary. This is especially prevalent in current state-of-the-art IL algorithms, such as dataset aggregation (DAgger) BID18 and generative adversarial imitation learning (GAIL) BID9. Although these approaches have been the dominant algorithms in IL, a major bottleneck for them is their reliance on high-quality demonstrations, which often require extensive supervision from human experts. In addition, a serious flaw in the learned policy π is its tendency to overfit to demonstration data, preventing it from generalizing to new ones. To overcome the aforementioned challenges in IL, a number of methods have been investigated to enhance the generalizability and data efficiency, or reduce the degree of human supervision. Initial efforts in this direction were based on the idea of meta learning BID6 BID8 ), in which the imitator is trained from a meta learner that is able to quickly learn a new task with only a few set of demonstrations. However, such schemes still require training the meta-learner with tremendous amount of time and demonstration data, leaving much room for improvement. Thus, a rapidly-growing body of literature based on the concept of using forward/inverse dynamics models to learn π within an environment in a self-supervised fashion BID0 BID11 BID13 has emerged in the past few years. One key advantage of the concept is that it provides an autonomous way for preparing training data, removing the need of human intervention. In this paper, we call it self-supervised IL.Self-supervised IL allows an imitator to collect training data by itself instead of using predefined extrinsic reward functions or expert supervision during training. It only needs demonstration during inference, drastically decreasing the time and effort required from human experts. Although the core principles of self-supervised IL are straightforward and have been exploited in many fields BID0 BID11 BID12, recent research efforts have been dedicated to addressing the challenges of multi-modality and multi-step planning. For example, the use of forward consistency loss and forward regularizer have been extensively investigated to enhance the task performance of the imitator BID0 BID13. This becomes especially essential when the lengths of trajectories grow and demonstration samples are sparse, as multiple paths may co-exist to lead the imitator from its initial observation to the goal observation. The issue of multi-step planning has also drawn a lot of attention from researchers, and is usually tackled by recurrent neural networks (RNNs) and step-by-step demonstrations BID11 BID13. The above self-supervised IL approaches report promising , however, most of them are limited in applicability due to several drawbacks. First, traditional methods of data collection are usually inefficient and time-consuming. Inefficient data collection in poor exploration, giving rise to a degradation in robustness to varying environmental conditions (e.g., noise in motor control) and generalizability to difficult tasks. Second, human bias in data sampling range tailored to specific interesting configurations is often employed BID0 BID11. Although a more general exploration strategy called curiosity-driven exploration was later proposed in BID12, it focuses only on exploration in states novel to the forward dynamics model, rather than those directly influential to the inverse dynamics model. Furthermore, it does not discuss the applicability to continuous control domains, and fails in high dimensional action spaces according to our experiments in Section 4. Unlike the approaches discussed above, we do not propose to deal with multi-modality or multi-step planning. Instead, we focus our attention on improving the overall quality of the collected samples in the context of self-supervised IL. This motivates us to equip the model with the necessary knowledge to explore the environment in an efficient and effective fashion. In this paper, we propose a straightforward and efficient self-supervised IL scheme, called adversarial exploration strategy, which motivates exploration of an environment in a self-supervised manner (i.e., without any extrinsic reward or human demonstration). Inspired by; BID23; BID24, we implement the proposed strategy by jointly training a deep reinforcement learning (DRL) agent and an inverse dynamics model competing with each other. The former explores the environment to collect training data for the latter, and receives rewards from the latter if the data samples are considered difficult. The latter is trained with the training data collected by the former, and only generates rewards when it fails to predict the true actions performed by the former. In such an adversarial setting, the DRL agent is rewarded only for the failure of the inverse dynamics model. Therefore, the DRL agent learns to sample hard examples to maximize the chances to fail the inverse dynamics model. On the other hand, the inverse dynamics model learns to be robust to the hard examples collected by the DRL agent by minimizing the probability of failures. As a , as the inverse dynamics model becomes stronger, the DRL agent is also incentivized to search for harder examples to obtain rewards. Overly hard examples, however, may lead to biased exploration and cause instability of the learning process. In order to stabilize the learning curve of the inverse dynamics model, we further propose a reward structure such that the DRL agent is encouraged to explore moderately hard examples for the inverse dynamics model, but refraining from too difficult ones for the latter to learn. The self-regulating feedback structure between the DRL agent and the inverse dynamics model enables them to automatically construct a curriculum for exploration. We perform extensive experiments to validate adversarial exploration strategy on multiple OpenAI gym BID3 robotic arm and hand manipulation task environments simulated by the MuJoCo physics engine , including FetchReach, FetchPush, FetchPickAndPlace, FetchSlide, and HandReach. These environments are intentionally selected by us for evaluating the performance of inverse dynamics model, as each of them allows only a very limited set of chained actions to transition the robotic arms and hands to target observations. We examine the effectiveness of our method by comparing it against a number of self-supervised IL schemes. The experimental show that our method is more effective and data-efficient than the other self-supervised IL schemes for both low-and high-dimensional observation spaces, as well as in environments with high-dimensional action spaces. We also demonstrate that in most of the cases the performance of the inverse dynamics model trained by our method is comparable to that directly trained with expert demonstrations. The above observations suggest that our method is superior to the other self-supervised IL schemes even in the absence of human priors. We further evaluate our method on environments with action space perturbations, and show that our method is able to achieve satisfactory success rates. To justify each of our design decisions, we provide a comprehensive set of ablative analysis and discuss their implications. The contributions of this work are summarized as follows:• We introduce an adversarial exploration strategy for self-supervised IL. It consists of a DRL agent and an inverse dynamics model developed for efficient exploration and data collection.• We employ a competitive scheme for the DRL agent and the inverse dynamics model, enabling them to automatically construct a curriculum for exploration of observation space.• We introduce a reward structure for the proposed scheme to stabilize the training process.• We demonstrate the proposed method and compare it with a number of baselines for multiple robotic arm and hand manipulation tasks in both low-and high-dimensional state spaces.• We validate that our method is generalizable to tasks with high-dimensional action spaces. The remainder of this paper is organized as follows. Section 2 introduces material. Section 3 describes the proposed adversarial exploration strategy in detail. Section 4 reports the experimental , and provides an in-depth ablative analysis of our method. Section 5 concludes. In this section, we briefly review DRL, policy gradient methods, as well as inverse dynamics model. DRL trains an agent to interact with an environment E. At each timestep t, the agent receives an observation x t ∈ X, where X is the observation space of E. It then takes an action a t from the action space A based on its current policy π, receives a reward r, and transitions to the next observation x. The policy π is represented by a deep neural network with parameters θ, and is expressed as π(a|x, θ). The goal of the agent is to learn a policy to maximize the discounted sum of rewards G t: DISPLAYFORM0 where t is the current timestep, γ ∈ the discount factor, and T the horizon. Policy gradient methods BID10 BID25 ) are a class of RL techniques that directly optimize the parameters of a stochastic policy approximator using policy gradients. Although these methods have achieved remarkable success in a variety of domains, the high variance of gradient estimates has been a major challenge. Trust region policy optimization (TRPO) BID21 circumvented this problem by applying a trust-region constraint to the scale of policy updates. However, TRPO is a second-order algorithm, which is relatively complicated and not compatible with architectures that embrace noise or parameter sharing BID22. In this paper, we employ a more recent family of policy gradient methods, called proximal policy optimization (PPO) BID22. PPO is an approximation to TRPO, which similarly prevents large changes to the policy between updates, but requires only first-order optimization. PPO is superior in its generalizability and sample complexity while retaining the stability and reliability of TRPO 1. An inverse dynamics model I takes as input a pair of observations (x, x), and predicts the actionâ required to reach the next observation x from the current observation x. It is formally expressed as: DISPLAYFORM0 where (x, x) are sampled from the collected data, and θ I represents the trainable parameters of I. During the training phase, θ I is iteratively updated to minimize the loss function L I, expressed as: DISPLAYFORM1 where d is a distance metric, and a the ground truth action. During the testing phase, a sequence of observations {x 0,x 1, · · ·,x T} is first captured from an expert demonstration. A pair of observations (x t,x t+1) is then fed into I at each timestep t. Starting fromx 0, the objective of I is to predict a sequence of actions {â 0,â 1, · · ·,â T −1} and transition the final observationx T as close as possible. In this section, we first describe the proposed adversarial exploration strategy. We then explain the training methodology in detail. Finally, we discuss a technique for stabilizing the training process. FIG0 shows a framework that illustrates the proposed adversarial exploration strategy, which includes a DRL agent P and an inverse dynamics model I. Assume that sequence of observations and actions generated by P as it explores E using a policy π. At each timestep t, P collects a 3-tuple training sample (x t, a t, x t+1) for I, while I predicts an actionâ t and generates a reward r t for P. In this work, I is modified from Eq. FORMULA1 to include an additional hidden vector h t, which recurrently encodes the information of the past observations. I is thus expressed as: where f (·) denotes the recurrent function. θ I is iteratively updated to minimize L I, formulated as: DISPLAYFORM0 where β is a scaling constant. We employ mean squared error β||a t −â t || 2 as the distance metric d(a t,â t), since we only consider continuous control domains in this paper. It can be replaced with a cross-entropy loss for discrete control tasks. We directly use L I as the reward r t for P, expressed as: DISPLAYFORM1 Our method targets at improving both the quality and efficiency of the data collection process performed by P, as well as the performance of I. Therefore, the goal of the proposed framework is twofold. First, P has to learn an adversarial policy π adv (a t |x t) such that its cumulated discounted rewards DISPLAYFORM2 ) is maximized. Second, I requires to learn an optimal θ I such that Eq. is minimized. Minimizing L I (i.e., r t) leads to decreased G t|π adv, forcing P to enhance π adv to explore more difficult samples to increase G t|π adv. This implies that P is motivated to focus on I's weak points, instead of randomly collecting ineffective training samples. Training I with hard samples not only accelerates its learning progress, but also helps to boost its performance. We describe the training methodology of our adversarial exploration strategy by a pseudocode presented in Algorithm 1. Assume that P's policy π adv is parameterized by a set of trainable parameters θ P, and is represented as π adv (a t |x t, θ P). We create two buffers Z P and Z I for storing the training samples of P and I, respectively. In the beginning, Z P, Z I, E, θ P, θ I, π adv, as well as a timestep cumulative counter c are initialized. A number of hyperparameters are set to appropriate values, including the number of iterations N iter, the number of episodes N episode, the horizon T, as well as the update period T P of θ P. At each timestep t, P perceives the current observation x t from E, takes an action a t according to π adv (a t |x t, θ P), and receives the next observation x t+1 and a termination indicator ξ (lines 9-11). ξ is set to 1 only when t equals T, otherwise it is set to 0. We then store (x t, a t, x t+1, ξ) and (x t, a t, x t+1) in Z P and Z I, respectively. We update θ P every T P timesteps using the samples stored in Z P, as shown in (lines 13-21). At the end of each episode, we update θ I with samples drawn from Z I according to the loss function L I defined in Eq. (line 23). Although adversarial exploration strategy is effective in collecting hard samples, it requires additional adjustments if P becomes too strong such that the collected samples are too difficult for I to learn. Overly difficult samples lead to a large variance in gradients derived from L I, which in turn cause a performance drop in I and instability in its learning process. We analyze this phenomenon in greater detail in Section 4.5. To tackle the issue, we propose a training technique that reshapes r t as follows: DISPLAYFORM0 Algorithm 1 Adversarial exploration strategy 1: Initialize Z P, Z I, E, and model parameters θ P & θ I 2: Initialize π adv (at|xt, θ P) 3: Initialize the timestep cumulative counter c = 0 4: Set Niter, N episode, T, and T P 5: for iteration i = 1 to Niter do 6:for episode e = 1 to N episode do for timestep t = 0 to T do 8:P perceives xt from E, and predicts an action at according to π adv (at|xt, θ P) xt+1 = E(xt, at)10: DISPLAYFORM0 Store (xt, at, xt+1, ξ) in Z P Store (xt, at, xt+1) in Z I if (c % T P) == 0 then Initialize an empty batch B Initialize a recurrent state ht for (xt, at, xt+1, ξ) in Z P do 17: Evaluateât = I(xt, xt+1|ht, θ I) (calculated from Eq. FORMULA4 18:Evaluate rt(xt, at, xt+1) = L I (at,ât|θ I) (calculated from Eq. FORMULA6 19: DISPLAYFORM0 Update θ P with the gradient calculated from the samples of B Reset Z P 22: DISPLAYFORM0 Update θ I with the gradient calculated from the samples of Z I (according to Eq. FORMULA5 24: end where δ is a pre-defined threshold value. This technique poses a restriction on the range of r t, driving P to gather moderate samples instead of overly hard ones. Note that the value of δ affects the learning speed and the final performance. We plot the impact of δ on the learning curve of I in Section 4.5. We further provide an example in our supplementary material to visualize the effect of this technique. In this section, we present experimental for a series of robotic tasks, and validate that (i) our method is effective in both low-and high-dimensional observation spaces; (ii) our method is effective in environments with high-dimensional action spaces; (iii) our method is more data efficient than the baseline methods; and (iv) our method is robust against action space perturbations. We first introduce our experimental setup. Then, we report experimental of robotic arm and hand manipulation tasks. Finally, we present a comprehensive set of ablative analysis to validate our design decisions. We first describe the environments and tasks. Next, we explain the evaluation procedure and the method for collecting expert demonstrations. We then walk through the baselines used for comparison. We evaluate our method on a number of robotic arm and hand manipulation tasks via OpenAI gym BID3 environments simulated by the MuJoCo physics engine. We use the Fetch and Shadow Dexterous Hand BID16 for the arm and hand manipulation tasks, respectively. For the arm manipulation tasks, which include FetchReach, FetchPush, FetchPickAndPlace, and FetchSlide, the imitator (i.e., the inverse dynamic model I) takes as inputs the positions and velocities of a gripper and a target object. It then infers the gripper's action in 3-dimensional space to manipulate it. For the hand manipulation task HandReach, the imitator takes as inputs the positions and velocities of the fingers of a robotic hand, and determines the velocities of the joints to achieve the goal. In addition to low-dimensional observations (i.e., position, velocity, and gripper state), we further perform experiments for the above tasks using visual observations (i.e., high-dimensional observations) in the form of camera images taken from a third-person perspective. The detailed description of the above tasks is specified in BID16. For the detailed configurations of these tasks, please refer to our supplementary material. The primary objective of our experiments is to demonstrate the efficiency of the proposed adversarial exploration strategy in collecting training data (in a self-supervised manner) for the imitator. We compare our strategy against a number of self-supervised data collection methods (referred to as "baselines" or "baseline methods") described in Section 4.1.4. As different baseline methods employ different data collection strategies, the learning curve of the imitator also varies for different cases. For a fair comparison, the model architecture of the imitator and the amount of training data are fixed for all cases. All of the experimental are evaluated and averaged over 20 trials, corresponding to 20 different random initial seeds. In each trial, we train an imitator by the training data collected by a single self-supervised data collection method. At the beginning of each episode, the imitator receives a sequence of observations {x 0,x 1, · · ·,x T} from a successful expert demonstration. At each timestep t, the imitator infers an actionâ t from an expert observationx t+1 and its current observation x t by Eq.. We periodically evaluate the imitator every 10K timesteps. The evaluation is performed by averaging the success rates of reachingx T over 500 episodes. The configuration of the imitator and the hyperparameters of the baselines are summarized in the supplementary material. For each task mentioned in Section 4.1.1, we first randomly configure task-relevant settings (e.g., goal position, initial state, etc.). We then collect demonstrations from non-trivial and successful episodes performed by a pre-trained expert agent. Please note that the collected demonstrations only contain sequences of observations. The implementation details of the expert agent and the method for filtering out trivial episodes are presented in our supplementary material. We compare our proposed methodology with the following four baseline methods in our experiments.• Random: This method collects training samples by random exploration. We consider it to be an important baseline because of its simplicity and prevalence in a number of research works on self-supervised IL BID0 BID11 BID13 ).• Demo: This method trains the imitator directly with expert demonstrations. It serves as the performance upper bound, as the training data is the same as the testing data for this method.• Curiosity: This method trains a DRL agent via curiosity BID12 to collect training samples. Unlike the original implementation, we replace its DRL algorithm with PPO, as training should be done on a single thread for a fair comparison with the other baselines. This is alo an important baseline due to its effectiveness in BID13.• Noise BID15: In this method, noise is injected to the parameter space of a DRL agent to encourage exploration BID15. Please note that its exploratory behavior relies entirely on parameter space noise, instead of using any extrinsic reward. We include this method due to its superior performance and data efficiency in many DRL tasks. We compare the performance of the proposed method and the baselines on the robotic arm manipulation tasks described in Section 4.1.1. As opposed to discrete control domains, these tasks are especially challenging, as the sample complexity grows in continuous control domains. Furthermore, the imitator may not have the complete picture of the environment dynamics, increasing its difficulty to learn an inverse dynamics model. In FetchSlide, for instance, the movement of the object on the slippery surface is affected by both friction and the force exerted by the gripper. It thus motivates us to investigate whether the proposed method can help overcome the challenge. In the subsequent paragraphs, we discuss the experimental in both low-and high-dimensional observation spaces, and plot them in Figs. 2 and 3, respectively. All of the are obtained by following the procedure described in Section 4.1.2. The shaded regions in Figs. 2 and 3 represent the confidence intervals. Low-dimensional observation spaces. FIG1 plots the learning curves for all of the methods in low-dimensional observation spaces. In all of the tasks, our method yields superior or comparable performance to the baselines except for Demo, which is trained directly with expert demonstrations. In FetchReach, it can be seen that every method achieves a success rate of 1.0. This implies that it does not require a sophisticated exploration strategy to learn an inverse dynamics model in an environment where the dynamics is relatively simple. It should be noted that although all methods reach the same final success rate, ours learns significantly faster than Demo. In contrast, in FetchPush, our method is comparable to Demo, and demonstrates superior performance to the other baselines. Our method also learns drastically faster than all the other baselines, which confirms that the proposed strategy does improve the performance and efficiency of self-supervised IL. Our method is particularly effective in tasks that require an accurate inverse dynamics model. In FetchPickAndPlace, for example, our method surpasses all the other baselines. However, all methods including Demo fail to learn a successful inverse dynamics model in FetchSlide, which suggests that it is difficult to train an imitator when the outcome of an action is not completely dependent on the action itself. It is worth noting that Curiosity loses to Random in FetchPush and FetchSlide, and Noise performs even worse than these two methods in all of the tasks. We therefore conclude that Curiosity is not suitable for continuous control tasks, and the parameter space noise strategy cannot be directly applied to self-supervised IL. In addition to the quantitative presented above, we further discuss the empirical qualitatively. Please refer our supplementary material for a description of the qualitative . High-dimensional observation spaces. FIG2 plots the learning curves of all methods in highdimensional observation spaces. It can be seen that our method performs significantly better than the other baseline methods in most of the tasks, and is comparable to Demo. In FetchPickAndPlace, our method is the only one that learns a successful inverse dynamics model. Similar to the in FIG1, Curiosity is no better than Random in high-dimensional observation spaces. Please note that we do not include Noise in FIG2 as it performs worse enough already in low-dimensional settings. FIG1 plots the learning curves for each of the methods considered. Please note that Curiosity, Noise and our method are pre-trained with 30K samples collected by random exploration, as we observe that these methods on their own suffer from large errors in an early stage during training, which prevents them from learning at all. After the first 30K samples, they are trained with data collected by their exploration strategy instead. From the in FIG1, it can be seen that Demo easily stands out from the other methods as the best-performing model, surpassing them all by a considerable extent. Although our method is not as impressive as Demo, it significantly outperforms all of the other baseline methods, achieving a success rate of 0.4 while the others are still stuck at around 0.2. The reason that the inverse dynamics models trained by the self-supervised data-collection strategies discussed in this paper (including ours and the other baselines) are not comparable to the Demo baseline in the HandReach task is primarily due to the high-dimensional action space. It is observed that the data collected by the self-supervised data-collection strategies only cover a very limited range of the state space in the HandReach environment. Therefore, the inverse dynamics models trained with these data only learn to imitate trivial poses, leading to the poor success rates presented in FIG1. We evaluate the performance of the imitator trained in an environment with action space perturbations to validate the robustness of our adversarial exploration strategy. In such an environment, every action taken by the DRL agent is perturbed by a Gaussian random noise, such that the training samples collected by the DRL agent are not inline with its actual intentions. Please note that we only inject noise during the training phase, as we aim to validate the robustness of the proposed data collection strategy. The scale of the injected noise is specified in the supplementary material. We report the performance change rates of various methods for different tasks in Table. 1. The performance change rate is defined as: DISPLAYFORM0, where P r perturb and P r orig represent the highest success rates with and without action space perturbations, respectively. From Table. 1, it can be seen that our method retains the performance for most of the tasks, indicating that our method is robust to action space perturbations during the training phase. Please note that although Curiosity and Noise also achieve a change rate of 0% in HandReach and FetchSlide, they are not considered robust due to their poor performance in the original environment FIG1. Another interesting observation is that our method even gains some performance from action space perturbations in FetchPush and HandReach, which we leave as one of our future directions. We thus conclude that our method is robust to action space perturbations during the training phase, making it a practical option in real-world settings. In this section, we provide a set of ablative analysis. We examine the effectiveness of our method by an investigation of the training loss distribution, the stabilization technique, and the influence of δ. Please note that the value of δ is set to 1.5 by default, as described in our supplementary material. Training loss distribution. FIG3 plots the probability density function (PDF) of L I (derived from Eq. FORMULA5) by kernel density estimation (KDE) for the first 2K training batches during the training phase. The vertical axis corresponds to the probability density, while the horizontal axis represents the scale of L I. The curves Ours (w stab) and Ours (w/o stab) represent the cases where the stabilization technique described in Section 3.3 is employed or not, respectively. We additionally plot the curve Random in FIG3 to highlight the effectiveness of our method. It can be observed that both Ours (w stab) and Ours (w/o stab) concentrate on notably higher loss values than Random. This observation implies that adversarial exploration strategy does explore hard samples for inverse dynamics model. We validate the proposed stabilization technique in terms of the PDF of L I and the learning curve of the imitator, and plot the in FIG3 and 5, respectively. FIG3, it can be observed that the modes of Ours (w stab) are lower than those of Ours (w/o stab) in most cases, implying that the stabilization technique indeed motivates the DRL agents to favor those moderately hard samples. We also observe that for each of the five cases, the mode of Ours (w stab) is close to the value of δ (plotted in a dotted line), indicating that our reward structure presented in Eq. does help to regulate L I (and thus r t) to be around δ. To further demonstrate the effectiveness of the stabilization technique, we compare the learning curves of Ours (w stab) and Ours (w/o stab) in FIG4. It is observed that for the initial 10K samples of the five cases, the success rates of Ours (w/o stab) are comparable to those of Ours (w stab). However, their performance degrade drastically during the rest of the training phase. This observation confirms that the stabilization technique does contribute significantly to our adversarial exploration strategy. Although most of the DRL works suggest that the rewards should be re-scaled or clipped within a range (e.g., from -1 to 1), the unbounded rewards do not introduce any issues during the training process of our experiments. The empirical rationale is that the rewards received by the DRL agent are regulated by Eq. FORMULA8 to be around δ, as described in Section 4.5 and depicted in FIG3. Without the stabilization technique, however, the learning curves of the inverse dynamics model degrade drastically (as illustrated in FIG1, even if the reward clipping technique is applied. Influence of δ. FIG5 compares the learning curves of the imitator for different values of δ. For instance, Ours(0.1) corresponds to δ = 0.1. It is observed that for most of the tasks, the success rates drop when δ is set to an overly high or low value (e.g., 100.0 or 0.0), suggesting that a moderate value of δ is necessary for the stabilization technique. The value of δ can be adjusted dynamically by the adaptive scaling technique presented in BID15, which is left as our future direction. From the analysis presented above, we conclude that the proposed adversarial exploration strategy is effective in collecting difficult training data for the imitator. The analysis also validates that our stabilization technique indeed leads to superior performance, and is capable of guiding the DRL agent to collect moderately hard samples. This enables the imitator to pursue a stable learning curve. In this paper, we presented an adversarial exploration strategy, which consists of a DRL agent and an inverse dynamics model competing with each other for self-supervised IL. The former is encouraged to adversarially collect difficult training data for the latter, such that the training efficiency of the latter is significantly enhanced. Experimental demonstrated that our method substantially improved the data collection efficiency in multiple robotic arm and hand manipulation tasks, and boosted the performance of the inverse dynamics model in both low-and high-dimensional observation spaces. In addition, we validated that our method is generalizable to environments with high-dimensional action spaces. Moreover, we showed that our method is robust to action space perturbations. Finally, we provided a set of ablative analysis to validate the effectiveness for each of our design decisions. In addition to the quantitative presented above, we further discuss the empirical qualitatively. Through visualizing the training progress, we observe that our method initially acts like Random, but later focuses on interacting with the object in FetchPush, FetchSlide, and FetchPickAndPlace. This phenomenon indicates that adversarial exploration strategy naturally gives rise to a curriculum that improves the learning efficiency, which resembles curriculum learning BID2. Another benefit that comes with the phenomenon is that data collection is biased towards interactions with the object. Therefore, the DRL agent concentrates on collecting interesting samples that has greater significance, rather than trivial ones. For instance, the agent prefers pushing the object to swinging the robotic arm. On the other hand, although Curiosity explores the environment very thoroughly in the beginning by stretching the arm into numerous different poses, it quickly overfits to one specific pose. This causes its forward dynamics model to keep maintaining a low error, making it less curious about the surroundings. Finally, we observe that the exploratory behavior of Noise does not change as frequently as ours, Random, and Curiosity. We believe that the method's success in the original paper BID15 ) is largely due to extrinsic rewards. In the absence of extrinsic rewards, however, the method becomes less effective and unsuitable for data collection, especially in self-supervised IL. We employ PPO BID22 as the RL agent responsible for collecting training samples because of its ease of use and good performance. PPO computes an update at every timestep that minimizes the cost function while ensuring the deviation from the previous policy is relatively small. One of the two main variants of PPO is a clipped surrogate objective expressed as: DISPLAYFORM0 whereÂ is the advantage estimate, and a hyperparameter. The clipped probability ratio is used to prevent large changes to the policy between updates. The other variant employs an adaptive penalty on KL divergence, given by: DISPLAYFORM1 where β is an adaptive coefficient adjusted according to the observed change in the KL divergence. In this work, we employ the former objective due to its better empirical performance. In the experiments, the inverse dynamics model I(x t, x t+1 |h t, θ I) of all methods employs the same network architecture. For low-dimensional observation setting, we use 3 Fully-Connected (FC) layers with 256 hidden units followed by tanh activation units. For high-dimensional observation setting, we use 3-layer Convolutional Neural Network (CNN) followed by relu activation units. The CNNs are configured as,, and, with each element in the 3-tuple denoting the number of output features, width/height of the filter, and stride. The features extracted by stacked CNNs are then fed forward to a FC with 512 hidden units followed by relu activation units. For both low-and high-dimensional observation settings, we use the architecture proposed in BID22. During training, we periodically update the DRL agent with a batch of transitions as described in Algorithm. 1. We split the batch into several mini-batches, and update the RL agent with these mini-batches iteratively. The hyperparameters are listed in Table. 2 (our method). Our baseline Curiosity is implemented based on the work BID13. The authors in BID13 propose to employ a curiosity-driven RL agent BID12 efficiency of data collection. The curiosity-driven RL agent takes curiosity as intrinsic reward signal, where curiosity is formulated as the error in an agents ability to predict the consequence of its own actions. This can be defined as a forward dynamics model: DISPLAYFORM0 whereφ(x) is the predicted feature encoding at the next timestep, φ(x) the feature vector at the current timestep, a the action executed at the current timestep, and θ F the parameters of the forward model f. The network parameters θ F is optimized by minimizing the loss function L F: DISPLAYFORM1 For low-and high-dimensional observation settings, we use the architecture proposed in BID22. The implementation of φ depends on the model architecture of the RL agent. For low-dimensional observation setting, we implement φ with the architecture of low-dimensional observation PPO. Note that φ does not share parameters with the RL agent in this case. For highdimensional observation setting, we share the features extracted by the CNNs of the RL agent, then feed these features to φ which consists of a FC with 512 hidden units followed by relu activation. The hyperparameters settings can be found in Table. 2(Curiosity). We directly apply the same architecture in BID15 without any modification. Please refer to BID15 for more detail.set of demonstrations in FetchReach is relatively difficult with only 100 episodes of demonstrations. A huge performance gap is observed when the number of episodes is increased to 1,000. Consequently, Demo is selected as our Demo baseline for the presentation of the experimental in Section 4. Another advantage is that Demo demands less memory than Demo. To test the robustness of our method to noisy actions, we add noise to the actions in the training stage. Letâ t denote the predicted action by the imitator. The actual noisy action to be executed by the robot is defined as:â t:=â t + N (0, σ), where σ is set as 0.01. Note thatâ t will be clipped in the range defined by each environment. In this section, we visualize the effects of our stabilization technique with a list of rewards r in FIG7. The rows of Before and After represent the rewards before and after reward shaping, respectively. The bar on the right-hand side indicates the scale of the reward. It can be observed in FIG7 that after reward shaping, the rewards are transformed to the negative distance to the specified δ (i.e., 2.5 in this figure). As a , our stabilization technique is able to encourage the DRL agent to pursue rewards close to δ, where higher rewards can be received.

represent the short summary for finding the corresponding scientific article (long).

------

The provided query could be "We prove the first-ever convergence proof of an asynchronous accelerated algorithm that attains a speedup." and the positive "	In this paper, we propose the Asynchronous Accelerated Nonuniform Randomized Block Coordinate Descent algorithm (A2BCD). We prove A2BCD converges linearly to a solution of the convex minimization problem at the same rate as NU_ACDM, so long as the maximum delay is not too large. This is the first asynchronous Nesterov-accelerated algorithm that attains any provable speedup. Moreover, we then prove that these algorithms both have optimal complexity. Asynchronous algorithms complete much faster iterations, and A2BCD has optimal complexity. Hence we observe in experiments that A2BCD is the top-performing coordinate descent algorithm, converging up to 4-5x faster than NU_ACDM on some data sets in terms of wall-clock time. To motivate our theory and proof techniques, we also derive and analyze a continuous-time analog of our algorithm and prove it converges at the same rate. In this paper, we propose and prove the convergence of the Asynchronous Accelerated Nonuniform Randomized Block Coordinate Descent algorithm (A2BCD), the first asynchronous Nesterovaccelerated algorithm that achieves optimal complexity. No previous attempts have been able to prove a speedup for asynchronous Nesterov acceleration. We aim to find the minimizer x * of the unconstrained minimization problem: DISPLAYFORM0 f (x) = f x,..., x (n) (FORMULA52 where f is σ-strongly convex for σ > 0 with L-Lipschitz gradient ∇f = (∇ 1 f, . . ., ∇ n f). x ∈ R d is composed of coordinate blocks x,..., x (n). The coordinate blocks of the gradient ∇ i f are assumed L i -Lipschitz with respect to the ith block. That is, ∀x, h ∈ R d: DISPLAYFORM1 where P i is the projection onto the ith block of R d. LetL 1 n n i=1 L i be the average block Lipschitz constant. These conditions on f are assumed throughout this whole paper. Our algorithm can also be applied to non-strongly convex objectives (σ = 0) or non-smooth objectives using the black box reduction techniques proposed in BID1. Hence we consider only the coordinate smooth, strongly-convex case. Our algorithm can also be applied to the convex regularized ERM problem via the standard dual transformation (see for instance): DISPLAYFORM2 Hence A2BCD can be used as an asynchronous Nesterov-accelerated finite-sum algorithm. Coordinate descent methods, in which a chosen coordinate block i k is updated at every iteration, are a popular way to solve equation 1.1. Randomized block coordinate descent updates a uniformly randomly chosen coordinate block i k with a gradient-descent-like step: DISPLAYFORM3 The complexity K of an algorithm is defined as the number of iterations required to decrease the error E(f (x k) − f (x *)) to less than (f (x 0) − f (x *)). Randomized coordinate descent has a complexity of K = O(n(L/σ) ln(1/)).Using a series of averaging and extrapolation steps, accelerated improves RBCD's iteration complexity K to O(n L /σ ln(1/)), which leads to much faster convergence whenL σ is large. This rate is optimal when all L i are equal. Finally, using a special probability distribution for the random block index i k, the non-uniform accelerated coordinate descent method BID2 (NU_ACDM) can further decrease the complexity to O(DISPLAYFORM4 L i /σ ln(1/)), which can be up to √ n times faster than accelerated RBCD, since some L i can be significantly smaller than L. NU_ACDM is the current state-of-the-art coordinate descent algorithm for solving equation 1.1.Our A2BCD algorithm generalizes NU_ACDM to the asynchronous-parallel case. We solve equation 1.1 with a collection of p computing nodes that continually read a shared-access solution vector y into local memory then compute a block gradient ∇ i f, which is used to update shared solution vectors (x, y, v). Proving convergence in the asynchronous case requires extensive new technical machinery. A traditional synchronous-parallel implementation is organized into rounds of computation: Every computing node must complete an update in order for the next iteration to begin. However, this synchronization process can be extremely costly, since the lateness of a single node can halt the entire system. This becomes increasingly problematic with scale, as differences in node computing speeds, load balancing, random network delays, and bandwidth constraints mean that a synchronous-parallel solver may spend more time waiting than computing a solution. Computing nodes in an asynchronous solver do not wait for others to complete and share their updates before starting the next iteration. They simply continue to update the solution vectors with the most recent information available, without any central coordination. This eliminates costly idle time, meaning that asynchronous algorithms can be much faster than traditional ones, since they have much faster iterations. For instance, random network delays cause asynchronous algorithms to complete iterations Ω(ln(p)) time faster than synchronous algorithms at scale. This and other factors that influence the speed of iterations are discussed in Hannah & Yin (2017a). However, since many iterations may occur between the time that a node reads the solution vector, and the time that its computed update is applied, effectively the solution vector is being updated with outdated information. At iteration k, the block gradient ∇ i k f is computed at a delayed iterateŷ k defined as 1: DISPLAYFORM5 for delay parameters j(k, 1),..., j(k, n) ∈ N. Here j(k, i) denotes how many iterations out of date coordinate block i is at iteration k. Different blocks may be out of date by different amounts, which is known as an inconsistent read. We assume 2 that j(k, i) ≤ τ for some constant τ < ∞.Asynchronous algorithms were proposed in to solve linear systems. General convergence and theory were developed later in BID5;;; Luo & Tseng (1992; 1993); There is also a rich body of work on asynchronous SGD. In the distributed setting, showed global convergence for stochastic variationally coherent problems even when the delays grow at a polynomial rate. , an asynchronous decentralized SGD was proposed with the same optimal sublinear convergence rate as SGD and linear speedup with respect to the number of workers. , authors obtained an asymptotic rate of convergence for asynchronous momentum SGD on streaming PCA, which provides insight into the tradeoff between asynchrony and momentum. , authors prove convergence for asynchronous SGD that highlight the tradeoff between faster iterations and iteration complexity. Further related work is discussed in Section 4. In this paper, we prove that A2BCD attains NU_ACDM's state-of-the-art iteration complexity to highest order for solving equation 1.1, so long as delays are not too large (see Section 2). The proof is very different from that of BID2, and involves significant technical innovations and complexity related to the analysis of asynchronicity. We also prove that A2BCD (and hence NU_ACDM) has optimal complexity to within a constant factor over a fairly general class of randomized block coordinate descent algorithms (see Section 2.1). This extends in to asynchronous algorithms with L i not all equal. Since asynchronous algorithms complete faster iterations, and A2BCD has optimal complexity, we expect A2BCD to be faster than all existing coordinate descent algorithms. We confirm with numerical experiments that A2BCD is the current fastest coordinate descent algorithm (see Section 5).We are only aware of one previous and one contemporaneous attempt at proving convergence for asynchronous Nesterov-accelerated algorithms. However, the first is not accelerated and relies on extreme assumptions, and the second obtains no speedup. Therefore, we claim that our are the first-ever analysis of asynchronous Nesterov-accelerated algorithms that attains a speedup. Moreover, our speedup is optimal for delays not too large 3.The work of Meng et al. claims to obtain square-root speedup for an asynchronous accelerated SVRG.In the case where all component functions have the same Lipschitz constant L, the complexity they obtain reduces to (n + κ) ln(1/) for κ = O τ n 2 (Corollary 4.4). Hence authors do not even obtain accelerated rates. Their convergence condition is τ < In a contemporaneous preprint, authors in skillfully devised accelerated schemes for asynchronous coordinate descent and SVRG using momentum compensation techniques. Although their complexity have the improved √ κ dependence on the condition number, they do not prove any speedup. Their complexity is τ times larger than the serial complexity. Since τ is necessarily greater than p, their imply that adding more computing nodes will increase running time. The authors claim that they can extend their to linear speedup for asynchronous, accelerated SVRG under sparsity assumptions. And while we think this is quite likely, they have not yet provided proof. We also derive a second-order ordinary differential equation (ODE), which is the continuous-time limit of A2BCD (see Section 3). This extends the ODE found in to an asynchronous accelerated algorithm minimizing a strongly convex function. We prove this ODE linearly converges to a solution with the same rate as A2BCD's, without needing to resort to the restarting techniques. The ODE analysis motivates and clarifies the our proof strategy of the main . We should consider functions f where it is efficient to calculate blocks of the gradient, so that coordinate-wise parallelization is efficient. That is, the function should be "coordinate friendly" Peng et al. (2016b). This is a very wide class that includes regularized linear regression, logistic regression, etc. The L 2 -regularized empirical risk minimization problem is not coordinate friendly in general, however the equivalent dual problem is, and hence can be solved efficiently by A2BCD (see , and Section 5).To calculate the k + 1'th iteration of the algorithm from iteration k, we use only one block of the gradient ∇ i k f. We assume that the delays j(k, i) are independent of the block sequence i k, but otherwise arbitrary (This is a standard assumption found in the vast majority of papers, but can be relaxed ; ;). Definition 1. Asynchronous Accelerated Randomized Block Coordinate Descent (A2BCD). Let f be σ-strongly convex, and let its gradient ∇f be L-Lipschitz with block coordinate Lipschitz parameters L i as in equation 1.2. We define the condition number κ = L/σ, and let L = min i L i. Using these parameters, we sample i k in an independent and identically distributed (IID) fashion according to DISPLAYFORM0 Let τ be the maximum asynchronous delay. We define the dimensionless asynchronicity parameter ψ, which is proportional to τ, and quantifies how strongly asynchronicity will affect convergence: DISPLAYFORM1 We use the above system parameters and ψ to define the coefficients α, β, and γ via eqs. (2.3) to (2.5). Hence A2BCD algorithm is defined via the iterations: eqs. (2.6) to (2.8). DISPLAYFORM2 See Section A for a discussion of why it is practical and natural to have the gradient DISPLAYFORM3 DISPLAYFORM4 Here we define y k = y 0 for all k < 0. The determination of the coefficients c i is in general a very involved process of trial and error, intuition, and balancing competing requirements. The algorithm doesn't depend on the coefficients, however; they are only an analytical tool. We define DISPLAYFORM5 To simplify notation 4, we assume that the minimizer x * = 0, and that f (x *) = 0 with no loss in generality. We define the Lyapunov function: DISPLAYFORM6 We now present this paper's first main contribution. Theorem 1. Let f be σ-strongly convex with a gradient ∇f that is L-Lipschitz with block Lipschitz constants DISPLAYFORM7 ). Then for A2BCD we have: DISPLAYFORM8 To obtain E[ρ k] ≤ ρ 0, it takes K A2BCD iterations for: 13) where O(·) is asymptotic with respect to σ −1/2 S → ∞, and uniformly bounded. DISPLAYFORM9 This is proven in Section B. A stronger for L i ≡ L can be proven, but this adds to the complexity of the proof; see Section E for a discussion. In practice, asynchronous algorithms are far more resilient to delays than the theory predicts. τ can be much larger without negatively affecting the convergence rate and complexity. This is perhaps because we are limited to a worst-case analysis, which is not representative of the average-case performance. Allen-Zhu et al. FORMULA15 (Theorem 5.1) shows a linear convergence rate of 1 − 2/ 1 + 2σ −1/2 S for NU_ACDM, which leads to the corresponding iteration complexity of DISPLAYFORM10. Hence, we have: DISPLAYFORM11 We can assume x * = 0 with no loss in generality since we may translate the coordinate system so that x * is at the origin. We can assume f (x *) = 0 with no loss in generality, since we can replace f (x) with f (x)−f (x *). Without this assumption, the Lyapunov function simply becomes: DISPLAYFORM12 Published as a conference paper at ICLR 2019 DISPLAYFORM13, the complexity of A2BCD asymptotically matches that of NU_ACDM. Hence A2BCD combines state-of-the-art complexity with the faster iterations and superior scaling that asynchronous iterations allow. We now present some special cases of the conditions on the maximum delay τ required for good complexity. Remark 1. Reduction to synchronous case. Notice that when τ = 0, we have ψ = 0, c i ≡ 0 and hence A k ≡ 0. Thus A2BCD becomes equivalent to NU_ACDM, the Lyapunov function 5 ρ k becomes equivalent to one found in BID2 (pg. 9), and Theorem 1 yields the same complexity. The maximum delay τ will be a function τ (p) of p, number of computing nodes. Clearly τ ≥ p, and experimentally it has been observed that τ = O(p). Let gradient complexity K(, τ) be the number of gradients required for an asynchronous algorithm with maximum delay τ to attain suboptimality. τ = 0, since with only 1 computing node there can be no delay. This corresponds to the serial complexity. We say that an asynchronous algorithm attains a complexity speedup if DISPLAYFORM0 is increasing in p. We say it attains linear complexity speedup if DISPLAYFORM1 In Theorem 1, we obtain a linear complexity speedup (for p not too large), whereas no other prior attempt can attain even a complexity speedup with Nesterov acceleration. In the ideal scenario where the rate at which gradients are calculated increases linearly with p, algorithms that have linear complexity speedup will have a linear decrease in wall-clock time. However in practice, when the number of computing nodes is sufficiently large, the rate at which gradients are calculated will no longer be linear. This is due to many parallel overhead factors including too many nodes sharing the same memory read/write bandwidth, and network bandwidth. However we note that even with these issues, we obtain much faster convergence than the synchronous counterpart experimentally. NU_ACDM and hence A2BCD are in fact optimal in some sense. That is, among a fairly wide class of coordinate descent algorithms A, they have the best-possible worst-case complexity to highest order. We extend the work in to encompass algorithms are asynchronous and have unequal L i. For a subset S ∈ R d, we let IC(S) (inconsistent read) denote the set of vectors v whose components are a combination of components of vectors in the set S. DISPLAYFORM0 Definition 4. Asynchronous Randomized Incremental Algorithms. Consider the unconstrained minimization problem equation 1.1 for function f satisfying the conditions stated in Section 1. We define the class A as algorithms G on this problem such that: DISPLAYFORM1 This is a rather general class: x k+1 can be constructed from any inconsistent reading of past iterates IC(X k), and any past gradient of an inconsistent read ∇ ij f (IC(X j)). DISPLAYFORM2 Hence A has a complexity lower bound: DISPLAYFORM3 Our proof in Section D follows very similar lines to;. In this section we present and analyze an ODE which is the continuous-time limit of A2BCD. This ODE is a strongly convex, and asynchronous version of the ODE found in. For simplicity, assume L i = L, ∀i. We rescale (I.e. we replace f (x) with 1 σ f.) f so that σ = 1, and hence κ = L/σ = L. Taking the discrete limit of synchronous A2BCD (i.e. accelerated RBCD), we can derive the following ODE 6 (see Section equation C.1): DISPLAYFORM0 We define the parameter η nκ 1/2, and the energy: DISPLAYFORM1. This is very similar to the Lyapunov function discussed in equation 2.11, with DISPLAYFORM2 the role of v k 2, and A k = 0 (since there is no delay yet). Much like the traditional analysis in the proof of Theorem 1, we can derive a linear convergence with a similar rate. See Section C.2. We may also analyze an asynchronous version of equation 3.1 to motivate the proof of our main theorem. HereŶ (t) is a delayed version of Y (t) with the delay bounded by τ. DISPLAYFORM0 Unfortunately, this energy satisfies (see Section equation C.4, equation C.7): DISPLAYFORM1 Hence this energy E(t) may not be decreasing in general. But, we may add a continuous-time asynchronicity error (see), much like in Definition 2, to create a decreasing energy. Let c 0 ≥ 0 and r > 0 be arbitrary constants that will be set later. Define: DISPLAYFORM2 Lemma 6. When rτ ≤ 1 2, the asynchronicity error A(t) satisfies: DISPLAYFORM3 DISPLAYFORM4 Hence f (Y (t)) convergence linearly to f (x *) with rate O exp −t/(nκ 1/2) Notice how this convergence condition is similar to Corollary 3, but a little looser. The convergence condition in Theorem 1 can actually be improved to approximately match this (see Section E).Proof. DISPLAYFORM5 The preceding should hopefully elucidate the logic and general strategy of the proof of Theorem 1. We now discuss related work that was not addressed in Section 1. Nesterov acceleration is a method for improving an algorithm's iteration complexity's dependence the condition number κ. FORMULA15 showed that many of the assumptions used in prior work (such as bounded delay τ < ∞) were unrealistic and unnecessary in general. In Hannah & Yin (2017a) the authors showed that asynchronous iterations will complete far more iterations per second, and that a wide class of asynchronous algorithms, including asynchronous RBCD, have the same iteration complexity as their synchronous counterparts. Hence certain asynchronous algorithms can be expected to significantly outperform traditional ones. authors propose a novel asynchronous catalyst-accelerated BID6 primal-dual algorithmic framework to solve regularized ERM problems. They structure the parallel updates so that the data that an update depends on is up to date (though the rest of the data may not be). However catalyst acceleration incurs a log(κ) penalty over Nesterov acceleration in general. In BID0, the author argues that the inner iterations of catalyst acceleration are hard to tune, making it less practical than Nesterov acceleration. To investigate the performance of A2BCD, we solve the ridge regression problem. Consider the following primal and corresponding dual objective (see for instance): DISPLAYFORM0 where A ∈ R d×n is a matrix of n samples and d features, and l is a label vector. We let A = [A 1, . . ., A m] where A i are the column blocks of A. We compare A2BCD (which is asynchronous accelerated), synchronous NU_ACDM (which is synchronous accelerated), and asynchronous RBCD (which is asynchronous non-accelerated). Nodes randomly select a coordinate block according to equation 2.1, calculate the corresponding block gradient, and use it to apply an update to the shared solution vectors. synchronous NU_ACDM is implemented in a batch fashion, with batch size p (1 block per computing node). Nodes in synchronous NU_ACDM implementation must wait until all nodes apply their computed gradients before they can start the next iteration, but the asynchronous algorithms simply compute with the most up-to-date information available. We use the datasets w1a (47272 samples, 300 features), wxa which combines the data from from w1a to w8a (293201 samples, 300 features), and aloi (108000 samples, 128 features) from. The algorithm is implemented in a multi-threaded fashion using C++11 and GNU Scientific Library with a shared memory architecture. We use 40 threads on two 2.5GHz 10-core Intel Xeon E5-2670v2 processors. See Section A.1 for a discussion of parameter tuning and estimation. The parameters for each algorithm are tuned to give the fastest performance, so that a fair comparison is possible. A critical ingredient in the efficient implementation of A2BCD and NU_ACDM for this problem is the efficient update scheme discussed in Lee & Sidford (2013b; a). In linear regression applications such as this, it is essential to be able to efficiently maintain or recover Ay. This is because calculating block gradients requires the vector A T i Ay, and without an efficient way to recover Ay, block gradient evaluations are essentially 50% as expensive as full-gradient calculations. Unfortunately, every accelerated iteration in dense updates to y k because of the averaging step in equation 2.6. Hence Ay must be recalculated from scratch. However Lee & Sidford (2013a) introduces a linear transformation that allows for an equivalent iteration that in sparse updates to new iteration variables p and q. The original purpose of this transformation was to ensure that the averaging steps (e.g. equation 2.6) do not dominate the computational cost for sparse problems. However we find a more important secondary use which applies to both sparse and dense problems. Since the updates to p and q are sparse coordinate-block updates, the vectors Ap, and Aq can be efficiently maintained, and therefore block gradients can be efficiently calculated. The specifics of this efficient implementation are discussed in Section A.2.In Table 5, we plot the sub-optimality vs. time for decreasing values of λ, which corresponds to increasingly large condition numbers κ. When κ is small, acceleration doesn't in a significantly better convergence rate, and hence A2BCD and async-RBCD both outperform sync-NU_ACDM since they complete faster iterations at similar complexity. Acceleration for low κ has unnecessary overhead, which means async-RBCD can be quite competitive. When κ becomes large, async-RBCD is no longer competitive, since it has a poor convergence rate. We observe that A2BCD and sync-NU_ACDM have essentially the same convergence rate, but A2BCD is up to 4 − 5× faster than sync-NU_ACDM because it completes much faster iterations. We observe this advantage despite the fact that we are in an ideal environment for synchronous computation: A small, homogeneous, high-bandwidth, low-latency cluster. In large-scale heterogeneous systems with greater synchronization overhead, bandwidth constraints, and latency, we expect A2BCD's advantage to be much larger. TAB4: Sub-optimality f (y k) − f (x *) (y-axis) vs time in seconds (x-axis) for A2BCD, synchronous NU_ACDM, and asynchronous RBCD for data sets w1a, wxa and aloi for various values of λ. An efficient implementation will have coordinate blocks of size greater than 1. This to ensure the efficiency of linear algebra subroutines. Especially because of this, the bulk of the computation for each iteration is computing ∇ i k f (ŷ k), and not the averaging steps. Hence the computing nodes only need a local copy of y k in order to do the bulk of an iteration's computation. Given this gradient ∇ i k f (ŷ k), updating y k and v k is extremely fast (x k can simply be eliminated). Hence it is natural to simply store y k and v k centrally, and update them when the delayed gradients ∇ i k f (ŷ k). Given the above, a write mutex over (y, v) has minuscule overhead (which we confirm with experiments), and makes the labeling of iterates unambiguous. This also ensures that v k and y k are always up to date when (y, v) are being updated. Whereas the gradient ∇ i k f (ŷ k) may at the same time be out of date, since it has been calculated with an outdated version of y k. However a write mutex is not necessary in practice, and does not appear to affect convergence rates or computation time. Also it is possible to prove convergence under more general asynchronicity. When defining the coefficients, σ may be underestimated, and L, L 1,..., L n may be overestimated if exact values are unavailable. Notice that x k can be eliminated from the above iteration, and the block gradient ∇ i k f (ŷ k) only needs to be calculated once per iteration. A larger (or overestimated) maximum delay τ will cause a larger asynchronicity parameter ψ, which leads to more conservative step sizes to compensate. To estimate ψ, one can first performed a dry run with all coefficient set to 0 to estimate τ. All function parameters can be calculated exactly for this problem in terms of the data matrix and λ. We can then use these parameters and this tau to calculate ψ. ψ and τ merely change the parameters, and do not change execution patterns of the processors. Hence their parameter specification doesn't affect the observed delay. Through simple tuning though, we found that ψ = 0.25 ed in good performance. In tuning for general problems, there are theoretical reasons why it is difficult to attain acceleration without some prior knowledge of σ, the strong convexity modulus BID3. Ideally σ is pre-specified for instance in a regularization term. If the Lipschitz constants L i cannot be calculated directly (which is rarely the case for the classic dual problem of empirical risk minimization objectives), the line-search method discussed in Section 4 can be used. As mentioned in Section 5, authors in Lee & Sidford (2013a) proposed a linear transformation of an accelerated RBCD scheme that in sparse coordinate updates. Our proposed algorithm can be given a similar efficient implementation. We may eliminate x k from A2BCD, and derive the equivalent iteration below: DISPLAYFORM0 where C and Q k are defined in the obvious way. Hence we define auxiliary variables p k, q k defined via: DISPLAYFORM0 These clearly follow the iteration: DISPLAYFORM1 Since the vector Q k is sparse, we can evolve variables p k, and q k in a sparse manner, and recover the original iteration variables at the end of the algorithm via A.1.The gradient of the dual function is given by: DISPLAYFORM2 As mentioned before, it is necessary to maintain or recover Ay k to calculate block gradients. Since Ay k can be recovered via the linear relation in equation A.1, and the gradient is an affine function, we maintain the auxiliary vectors Ap k and Aq k instead. Hence we propose the following efficient implementation in Algorithm 1. We used this to generate the in Table 5. We also note also that it can improve performance to periodically recover v k and y k, reset the values of p k, q k, and C to v k, y k, and I respectively, and restarting the scheme (which can be done cheaply in time O(d)).We let B ∈ R 2×2 represent C k, and b represent B −1. ⊗ is the Kronecker product. Each computing node has local outdated versions of p, q, Ap, Aq which we denotep,q,Âp,Âq respectively. We also find it convenient to define: DISPLAYFORM3 Algorithm 1 Shared-memory implementation of A2BCD Randomly select block i via equation 2.1. Read shared data into local memory:p ← p,q ← q,Âp ← Ap,Âq ← Aq,B ← B. Compute block gradient: DISPLAYFORM0 10: DISPLAYFORM1 11: DISPLAYFORM0 12: DISPLAYFORM1 Increase iteration count: k ← k + 1 14: end while 15: Recover original iteration variables: DISPLAYFORM2 We first recall a couple of inequalities for convex functions. Lemma 7. Let f be σ-strongly convex with L-Lipschitz gradient. Then we have: DISPLAYFORM0 We also find it convenient to define the norm: DISPLAYFORM1 B.1 Starting point First notice that using the definition equation 2.8 of v k+1 we have: DISPLAYFORM0 We have the following general identity: DISPLAYFORM1 It can also easily be verified from equation 2.6 that we have: DISPLAYFORM2 DISPLAYFORM3 This inequality is our starting point. We analyze the terms on the second line in the next section. To analyze these terms, we need a small lemma. This lemma is fundamental in allowing us to deal with asynchronicity. Lemma 8. Let χ, A > 0. Let the delay be bounded by τ. Then: DISPLAYFORM0 Proof. See Hannah & Yin (2017a). We have: DISPLAYFORM0 The terms in bold in equation B.8 and equation B.9 are a of the asynchronicity, and are identically 0 in its absence. Our strategy is to separately analyze terms that appear in the traditional analysis of Nesterov FORMULA15, and the terms that from asynchronicity. We first prove equation B.8: FIG5 equation B.10 follows from strong convexity (equation B.2 with x = y k and y = x *), and the fact that ∇f is L-Lipschitz. The term due to asynchronicity becomes: DISPLAYFORM0 DISPLAYFORM1 using Lemma 8 with χ = κψ −1, A = y k. Combining this with equation B.10 completes the proof of equation B.8.We now prove equation B.9: DISPLAYFORM2 Here the last line follows from Lemma 8 with χ = κψ DISPLAYFORM3 We can complete the proof using the following identity that can be easily obtained from equation 2.6: DISPLAYFORM4 Much like , we need a f (x k) term in the Lyapunov function (see the middle of page 357). However we additionally need to consider asynchronicity when analyzing the growth of this term. Again terms due to asynchronicity are emboldened. Lemma 10. We have: DISPLAYFORM0 Proof. From the definition equation 2.7 of x k+1, we can see that x k+1 − y k is supported on block i k. Since each gradient block ∇ i f is L i Lipschitz with respect to changes to block i, we can use equation B.1 to obtain: DISPLAYFORM1 Here the last line followed from the definition equation B.3 of the norm · * 1/2. We now analyze the middle term: DISPLAYFORM2 We then apply Lemma 8 to this with χ = 2h DISPLAYFORM3 Finally to complete the proof, we combine equation B.11, with equation B.12. The previous inequalities produced difference terms of the form y k+1−j − y k−j 2. The following lemma shows how these errors can be incorporated into a Lyapunov function. Lemma 11. Let 0 < r < 1 and consider the asynchronicity error and corresponding coefficients: DISPLAYFORM0 Remark 2. Interpretation. This means that an asynchronicity error term A k can negate a series of difference terms − ∞ j=1 s j y k+1−j − y k−j 2 at the cost of producing an additional error c 1 E k y k+1 − y k 2, while maintaining a convergence rate of r. This essentially converts difference terms, which are hard to deal with, into a y k+1 − y k 2 term which can be negated by other terms in the Lyapunov function. The proof is straightforward. Proof. DISPLAYFORM0 Noting the following completes the proof: DISPLAYFORM1 Given that A k allows us to negate difference terms, we now analyze the cost c 1 E k y k+1 − y k 2 of this negation. We have: DISPLAYFORM0 Proof. DISPLAYFORM1 Here equation B.13 following from equation 2.8, the definition of v k+1. equation B.14 follows from the inequality x + y 2 ≤ 2 x 2 + 2 y 2. The rest is simple algebraic manipulation. DISPLAYFORM2 (definitions of h and α: equation 2.3, and equation 2.5) = 1 DISPLAYFORM3 Rearranging the definition of ψ, we have: DISPLAYFORM4 Using this on equation B.15, we have: DISPLAYFORM5 This completes the proof. We are finally in a position to bring together all the all the previous together into a master inequality for the Lyapunov function ρ k (defined in equation 2.11). After this lemma is proven, we will prove that the right hand size is negative, which will imply that ρ k linearly converges to 0 with rate β. Lemma 13. Master inequality. We have: DISPLAYFORM0 Proof. DISPLAYFORM1 We now collect and organize the similar terms of this inequality. DISPLAYFORM2 Now finally, we add the function-value and asynchronicity terms to our analysis. We use Lemma 11 is with r = 1 − σ 1/2 S −1, and DISPLAYFORM3 Notice that this choice of s i will recover the coefficient formula given in equation 2.9. Hence we have: DISPLAYFORM4 (Lemmas 11 and 12) + c 1 2α In the next section, we will prove that every coefficient on the right hand side of equation B.16 is 0 or less, which will complete the proof of Theorem 1. DISPLAYFORM5 DISPLAYFORM6 Here the last line followed since ψ ≤ 1 2 and σ 1/2 S −1 ≤ 1. We now analyze the coefficient of DISPLAYFORM7 Proof. DISPLAYFORM8 in Lemma 13 is non-positive. Proof. We first need to bound c 1.(equation B.18 and equation 2.9) c 1 = s DISPLAYFORM9 It can be easily verified that if x ≤ 1 2 and y ≥ 0, then (1 − x) −y ≤ exp(2xy). Using this fact with x = σ 1/2 S −1 and y = τ, we have: DISPLAYFORM10 (since ψ ≤ 3/7 and hence τ σ DISPLAYFORM11 We now analyze the coefficient of ∇f (ŷ k) DISPLAYFORM12 Proof. DISPLAYFORM13 Here the last inequality follows since β ≤ 1 and α ≤ σ 1/2 S −1. We now rearrange the definition of ψ to yield the identity: DISPLAYFORM14 Using this, we have: DISPLAYFORM15 Here the last line followed since L ≤ L, ψ ≤ 3 7, and τ ≥ 1. Hence the proof is complete. Proof of Theorem 1. Using the master inequality 13 in combination with the previous Lemmas 14, 15, 16, and 17, we have: DISPLAYFORM16 When we have: DISPLAYFORM17 then the Lyapunov function ρ k has decreased below ρ 0 in expectation. Hence the complexity K satisfies: DISPLAYFORM18 Now it can be shown that for 0 < x ≤ 1 2, we have: DISPLAYFORM19 Since n ≥ 2, we have σ 1/2 S −1 ≤ 1 2. Hence: DISPLAYFORM20 An expression for K NU_ACDM , the complexity of NU_ACDM follows by similar reasoning. DISPLAYFORM21 Finally we have: DISPLAYFORM22 which completes the proof. C.1 Derivation of ODE for synchronous A2BCDIf we take expectations with respect to E k, then synchronous (no delay) A2BCD becomes: DISPLAYFORM0 We find it convenient to define η = nκ 1/2. Inspired by this, we consider the following iteration: DISPLAYFORM1 C.1 Derivation of ODE for synchronous A2BCDfor coefficients: DISPLAYFORM2 s is a discretization scale parameter that will be sent to 0 to obtain an ODE analogue of synchronous A2BCD. We first use equation DISPLAYFORM3 The proof of convergence is completed in Section 3. For parameter set σ, L 1,..., L n, n, we construct a block-separable function f on the space R As mentioned, a stronger than Theorem 1 is possible. In the case when L i = L for all i, we can consider a slight modification of the coefficients: DISPLAYFORM0 (E.1) DISPLAYFORM1 for the asynchronicity parameter: DISPLAYFORM2 This leads to complexity: DISPLAYFORM3 Here there is no restriction on ψ as in Theorem 1, and hence there is no restriction on τ. Assuming ψ ≤ 1 gives optimal complexity to within a constant factor. Notice then that the ing condition of τ τ ≤ 1 6 nκ −1/2 (E.9) now essentially matches the one in Theorem 3 in Section 3. While this is stronger, it increases the complexity of the proof substantially. So in the interests of space and simplicity, we do not prove this stronger ."

A simple yet effective imitation learning scheme that incentivizes exploration of an environment without any extrinsic reward or human demonstration.

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: This paper proposes a dual variational autoencoder (DualVAE), a framework for generating images corresponding to multiclass labels. Recent research on conditional generative models, such as the Conditional VAE, exhibit image transfer by changing labels. However, when the dimension of multiclass labels is large, these models cannot change images corresponding to labels, because learning multiple distributions of the corresponding class is necessary to transfer an image. This leads to the lack of training data. Therefore, instead of conditioning with labels, we condition with latent vectors that include label information. DualVAE divides one distribution of the latent space by linear decision boundaries using labels. Consequently, DualVAE can easily transfer an image by moving a latent vector toward a decision boundary and is robust to the missing values of multiclass labels. To evaluate our proposed method, we introduce a conditional inception score (CIS) for measuring how much an image changes to the target class. We evaluate the images transferred by DualVAE using the CIS in CelebA datasets and demonstrate state-of-the-art performance in a multiclass setting. Recent conditional generative models have shown remarkable success in generating and transferring images. Specifically, a conditional variational autoencoder (CVAE) BID4 can generate conditional images by learning the latent space Z that corresponds to multiclass labels. In addition, StarGAN BID1 and FaderNetworks BID5 can generate images corresponding to multiple domains by conditioning with domains such as attributes. However, when the dimension of the multiclass is increased, these models cannot transfer the images corresponding to one arbitrary domain (an element of a multiclass label). The possible reasons are the following. For simplicity, we consider a binary multiclass classification. To transfer an image of a certain class, it is necessary to learn the distributions of the corresponding class. That is, assuming that the number of classes in the multiclass is N, conditional models need to create 2 N distributions. However, when N is large, training is difficult as O(2 N) training samples will be required. Hence, instead of conditioning with labels, we propose DualVAE, which conditions with latent vectors that include label information. DualVAE divides one distribution of the latent space by N linear decision boundaries which need to learn only O(N) parameters by adding another decoder p w (y|z) to a variational autoencoder (VAE) BID3. DualVAE assumes that a label is a linear combination of vectors of the latent space and the dual latent space. There are two advantages to the DualVAE decoder p w (y|z) being a linear model. First, DualVAE can easily transfer an image by moving a latent vector toward a decision boundary. Next, DualVAE is robust to the missing values of multiclass labels. In addition to this method, we propose the conditional inception score (CIS), a new metric for conditional transferred images. Although the evaluation methods often used in the generation models are the Inception Score (IS) BID9 and the Fréchet Inception Distance BID2, they are used for evaluating the diversity of images and not suitable for evaluating transferred images conditioned with domains such as attributes or classes. Therefore, we propose a new metric to evaluate two properties: the first property pertains to whether images in one domain are transferred properly to images in another domain; the second property pertains to whether images in one domain Figure 1: Conditional VAE learns 2 n distributions for each binary multiclass label when the number of class is n. DualVAE learns n decision boundaries for dividing a distribution of latent space. u 1 is a parameter of a decision boundary, which we call a dual vector.transferred to images in another domain can preserve the original properties. By using the CIS, we compare DualVAE with other methods that can perform image-to-image translations for multiple domains. In summary, the contributions from this study are as follows: 1) We introduce DualVAE, a method for transferring images corresponding to multiclass labels and demonstrate that images can be transferred quantitatively and qualitatively. 2) We propose the CIS, a new metric that can evaluate transferred images corresponding to multiclass labels. Conditional model Several studies have been conducted to generate or transfer images conditioned with labels. For example, conditional VAE BID4 is an extension of a VAE BID3 where latent variables z are inferred using image x and label y, and image x is reconstructed with y,z. Further, a CGAN BID7 ) is a conditional model using a GAN, where a noise z and a class label y are input to the generator, and learning is performed similarly to the GAN using image x corresponding to class label y. FaderNetworks BID5 learns latent variables from which label information is eliminated by using adversarial learning and assigns attributes to images by providing labels to the decoder. Furthermore, StarGAN BID1, a method of domain transfer, had succeeded in outputting a beautiful image corresponding to an attribute by conditioning with a domain (attribute). However, all these methods are models conditioned with labels; therefore, as the dimension of the labels becomes larger, the number of training samples becomes insufficient. Connection to the Information Bottleneck As with DualVAE, there are several papers related to finding a latent variable z that predicts label y. For example, Information Bottleneck (IB) BID11 is a method for obtaining a latent expression z that solves task y. IB is a method which leaves the latent information z for solving the task y by maximizing the mutual information amount I(Z; Y). At the same time, extra information about input x is discarded by minimizing I(Z; X). Variational Information Bottleneck (VIB) BID0 succeeded in parameterizing the IB with a neural network, by performing a variational approximation. VIB can also be considered as a kind of extension of VAE. VAE minimizes the mutual information I(Z; i) between individual data i and latent variable z while maximizing I(Z; X). DualVAE can be regarded as a framework of VIB as well, and it minimizes I(Z; i) while maximizing I(Z; Y) and I(Z; X). We can also regard DualVAE as a probabilistic matrix factorization (PMF) BID8 extended to a generative model. A PMF is used in several application areas, primarily in collaborative filtering, which is a typical recommendation algorithm. It can predict missing ratings of users by assuming that the user's ratings are modeled by a linear combination of the item and user latent factors. Similarly, we experimentally show that DualVAE is also robust to missing labels. We devised DualVAE by adding a decoder p w (y|z) = p(y|z, u) to the VAE to learn the decision boundaries between classes. Here, z is a vector of the latent space Z, u is a vector of the dual latent space Z * and y is a label. Unlike the CVAE, this model does not require label y at the time of inference of z corresponding to x, and the difference is shown in FIG0. The objective function of the VAE is as follows: DISPLAYFORM0 where φ, θ are parameters of the encoder and decoder of the VAE, respectively. The lower bound of DualVAE is as follows: DISPLAYFORM1 where p w (y|z) = Bern(y|σ(Uz)). Here, U is a domain feature matrix whose row vector is a dual vector u and Bern is a Bernoulli distribution. As you can see from Equation 2, the objective function of DualVAE is the objective function of the VAE plus the expectation of log-likelihood of p w (y|z) Specifically, training is performed such that the inner product of z j ∈ Z and u i ∈ Z * predicts the label y ij where j is the index of a sample and i is the index of a domain. At the same time, we find the values of θ and φ that maximize the lower bound in Equation 1.We transfer the images on domain i by performing the following operation. We calculated the following vector w i: DISPLAYFORM2 where λ(∈ R) is a parameter. Image transfer can be demonstrated by changing λ and decoding w i. Equation 3 corresponds to moving a latent vector toward a decision boundary. Require: images (x j) m j=1, batch size M, indicator function I ij VAE/encoder optimizers: g, g e, hyper parameter α, and the label matrix Y = (y ij). Initialize encoder parameter, decoder parameter and dual vector: θ, φ, U = (u i) DISPLAYFORM0 Although IS BID9 ) is a score for measuring generated images, it can only measure the diversity of the images, and cannot be used for evaluating the domain transfer of the images. Therefore, we proposed using a CIS, a score for evaluating the transformation of images into multiclass target domains. The CIS is a scalar value calculated from the sum of two elements. The first is whether the domain transfer of the original image has been successful (transfer score), and the second is whether the features other than the transferred domain are retained (reconstruction score). The computation flow of these scores can be found in FIG1.We calculated the CIS using Algorithm 2. First, we assumed that the number of domains is n and the domain that each image belonged to was known. We finetuned Inception-v3 BID10 using train images as inputs and domains as outputs. To enable the model to classify the images with the domains, we replaced the last layer of the model with a new layer that had n outputs. Next, we transferred test images into n domain images and loaded the transferred images into the pretrained Inception-v3. Through this process, we obtained an n × n matrix for every original image because one image was transferred into n domain images and each domain image was mapped to an n-dimension vector. We subsequently mapped the original image into an n-dimension vector using Inception-v3 and subtracted this vector from each row of the n × n matrix. We named this matrix M. The key points are the following: the diagonal elements of M should be large because the specified domain should be changed significantly, and the off-diagonal elements of M should be small because the transferred images should preserve the original features. DISPLAYFORM0 In the algorithm, abs denotes taking the absolute value, diag denotes taking the diagonal elements of the matrix, notdiag denotes taking the nondiagonal elements, avg denotes taking the mean of the multiclass values. x is an original image, and x (i) denotes a transferred image on domain i. We performed a standard image transfer task with the 40 attributes in the CelebA BID6 dataset, which comprises approximately 200,000 images of faces of celebrities. Comparison of DualVAE and several models DualVAE was compared with several models capable of performing image-to-image translations for multiclass labels using a single model. In each model, we calculated the CIS several times when applying Algorithm 2 on 160 CelebA test images; subsequently, the average and standard deviation were obtained. DualVAE obtained a higher CIS than the other models and the are shown in TAB0 and Figure 4. Robustness to sparsity To demonstrate experimentally that DualVAE is robust to the missing values of multiclass labels, the following steps were performed. We calculated the rs and ts values when applying Algorithm 2 on 160 CelebA test images and plotted the figure below when we changed the missing ratio of CelebA's domain labels and the λ in Equation 3. StarGAN (b) s = 0.9. All identical images were generated, and image transfer was not conducted properly. As shown in FIG3, DualVAE is robust in terms of the sparseness of domain labels, and the CIS does not decrease even when 90% of the labels are missing. Meanwhile, we found that StarGAN is not as robust as DualVAE with respect to sparseness. When 90% of the domain labels are missing, StarGAN cannot learn at all and generates identical images. We proposed DualVAE, a simple framework for generating and transferring images corresponding to multiclass labels. Further, we introduced the CIS, a new metric for measuring how much of an image corresponding to the change of labels could be generated. The decoder of DualVAE was a simple linear model in this study; however, we would like to test more complex models in the future.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

a new framework using dual space for generating images corresponding to multiclass labels when the number of class is large

We propose a novel, projection based way to incorporate the conditional information into the discriminator of GANs that respects the role of the conditional information in the underlining probabilistic model.