Question:
represent the scientific article to find a short tldr.

For example, 'Hashing-based collaborative filtering learns binary vector representations (hash codes) of users and items, such that recommendations can be computed very efficiently using the Hamming distance, which is simply the sum of differing bits between two hash codes. A problem with hashing-based collaborative filtering using the Hamming distance, is that each bit is equally weighted in the distance computation, but in practice some bits might encode more important properties than other bits, where the importance depends on the user. To this end, we propose an end-to-end trainable variational hashing-based collaborative filtering approach that uses the novel concept of self-masking: the user hash code acts as a mask on the items (using the Boolean AND operation), such that it learns to encode which bits are important to the user, rather than the user's preference towards the underlying item property that the bits represent. This allows a binary user-level importance weighting of each item without the need to store additional weights for each user. We experimentally evaluate our approach against state-of-the-art baselines on 4 datasets, and obtain significant gains of up to 12% in NDCG. We also make available an efficient implementation of self-masking, which experimentally yields <4% runtime overhead compared to the standard Hamming distance. Collaborative filtering is an integral part of personalized recommender systems and works by modelling user preference on past item interactions to predict new items the user may like . Early work is based on matrix factorization approaches that learn a mapping to a shared m-dimensional real-valued space between users and items, such that user-item similarity can be estimated by the inner product. The purpose of hashing-based collaborative filtering is the same as traditional collaborative filtering, but allows for fast similarity searches to massively increase efficiency (e.g., realtime brute-force search in a billion items ). This is done by learning semantic hash functions that map users and items into binary vector representations (hash codes) and then using the Hamming distance (the sum of differing bits between two hash codes) to compute user-item similarity. This leads to both large storage reduction (floating point versus binary representations) and massively faster computation through the use of the Hamming distance. One problem with hashing-based collaborative filtering is that each bit is weighted equally when computing the Hamming distance. This is a problem because the importance of each bit in an item hash code might differ between users. The only step towards addressing this problem has been to associate a weight with k-bit blocks of each hash code . However, smaller values of k lead to increased storage cost, but also significantly slower computation due to the need of computing multiple weighted Hamming distances. To solve this problem, without using any additional storage and only a marginal increase in computation time, we present Variational Hashing-based collaborative filtering with Self-Masking (VaHSM-CF). VaHSM-CF is our novel variational deep learning approach for hashing-based collaborative filtering that learns hash codes optimized for selfmasking. Self-masking is a novel technique that we propose in this paper for user-level bit-weighting on all items. Self-masking modifies item hash codes by applying an AND operation between an item and user hash code, before computing the standard Hamming distance between the user and selfmasked item hash codes. Hash codes optimized with self-masking represent which bit-dimensions encode properties that are important for the user (rather than a bitwise -1/1 preference towards each property). In practice, when ranking a set of items for a specific user, self-masking ensures that only bit differences on bit-dimensions that are equal to 1 for the user hash code are considered, while ignoring the ones with a -1 value, thus providing a user-level bitwise binary weigthing. Since selfmasking is applied while having the user and item hash codes in the lowest levels of memory (i.e., register), it only leads to a very marginal efficiency decrease. We contribute (i) a new variational hashing-based collaborative filtering approach, which is optimized for (ii) a novel self-masking technique, that outperforms state-of-the-art baselines by up to 12% in NDCG across 4 different datasets, while experimentally yielding less than 4% runtime overhead compared to the standard Hamming distance. We publicly release the code for our model, as well as an efficient implementation of the Hamming distance with self-masking 1. We focus on collaborative filtering with explicit feedback, which assumes that users and items are related via a user-specified rating: the task is to rank a pool of pre-selected items. This is different from implicit feedback, where the task is to estimate the pool of items that are of interest to the user. Matrix factorization is one of the most successful collaborative filtering methods , but to reduce storage requirements and speed up computation, hashing-based collaborative filtering has been researched. For hashing-based methods the users and items are represented as binary hash codes (as opposed to real-valued vectors), such that the highly efficient Hamming distance (as opposed to the inner product) can be used for computing user-item similarities. Two-stage approaches. Early hashing-based collaborative filtering methods include two stages: First, real-valued user and item vectors are learned, and then the real-valued vectors are transformed into binary hash codes. employ matrix factorization initially, followed by a binary quantization of rounding the real-valued vectors, while ensuring that the hash code is preference preserving of the observed user-item ratings using their proposed Constant Feature Norm constraint. and both explore binary quantization strategies based on orthogonal rotations of the real-valued vectors, which share similarities with Spectral Clustering . However, the two-stage approaches often suffer from large quantization errors , because the hash codes are not learned directly, but rather based on different quantization procedures. Learned hashing approaches. propose Discrete Collaborative Filtering (DCF), which is a binary matrix factorization approach that directly learns the hash codes using relaxed integer optimization, while enforcing bit balancing and decorrelation constraints. Extensions of DCF have focused on incorporating side-information (e.g., reviews associated with a rating); ) and have been redesigned for implicit feedback signals . More recent work addresses the problem that hashing-based collaborative filtering methods have reduced representational power compared to real-valued vectors, but increasing the hash code dimensionality to match the amount of bits used in the real-valued case hurts model generalization . To address this, propose Compositional Coding for Collaborative Filtering (CCCF), which is broadly similar to learning compositional codes for (word) embedding compression . CCCF is a hybrid approach that combines hash codes and real-valued weights: each hash code is split into k blocks of r bits each, and each block is associated with a real-valued scalar indicating the weight of the block. The distance between two CCCF hash codes is then computed as a weighted sum of the Hamming distances of the individual blocks, where each weight is the product of each block's weight. The problem with this approach is that each block requires an individual Hamming distance computation, as well as floating point multiplications of the block weights. In fact, the CCCF block construction no longer allows for highly efficient Boolean operations because the distance computation is weighted by each block's weight. Another problem with CCCF is that it drastically increases storage requirements by needing to store the real-valued weights for all blocks in a hash code. In contrast to CCCF, our proposed variational hashing-based collaborative filtering with selfmasking solves the same problems, as it effectively allows to disable unimportant bits -corresponding to a 1-bit block size with 0/1 weights -without needing to store any additional weights or vectors. Additionally, after having applied the self-masking, user-item similarity can still be computed using only a single Hamming distance on the two hash codes. Hashing-based Collaborative filtering aims to learn binary user and item representations (called hash codes), such that the distance between the representations indicates how well user u likes item i. In practice, the Hamming distance is used due to its fast hardware-level implementation. Formally, we learn z u ∈ {−1, 1} m and z i ∈ {−1, 1} m, where m is the number of bits in the hash code, which is typically chosen to fit into a machine word. The preference of user u for item i is specified by the rating R u,i ∈ {1, 2, 3, ..., K}, where K is the maximum rating, such that the Hamming distance between z u and z i is low when R u,i is high. Computing the Hamming distance computation is extremely efficient due to fast hardware-level implementation of the Boolean operations, as where SUM is computed fast on hardware using the popcnt instruction. Given a user and set of items, the integer-valued Hamming distances can be linear-time sorted using e.g. radix sort (because Hamming distances are bounded in [0, m]) in ascending order to create a ranked list based on user preference . The Hamming distance assigns equal weights to all bits, but in reality bit importance might differ among users. For example, if we consider each bit an encoding of a specific property of an item (e.g., a movie being a thriller), then the weight of each property would be dependent on each user's preference. However, since the hash codes are binary, it is not possible to encode such preference weights without using more storage and computation time due to no longer operating on binary values. In fact, no existing method even allows disabling specific bits for certain users (corresponding to the case of 0 preference weight). We next present a solution for the latter problem, which encodes the importance of each bit directly into the user hash code, and therefore does not require any additional storage. We define the Hamming distance with self-masking: We first apply an AND operation between the user and item hash codes, and then compute the Hamming distance between that and the user hash code. This fundamentally changes the purpose of the user hash code: instead of encoding a positive or negative preference for a property, it encodes which properties are important to the user (-1's from the user hash code are copied to the item due to the AND operation). This allows the model to disable unimportant bits on a user-level, meaning that we enable the model to produce user-specific item representations while still only storing a single hash code for each user and item respectively. Self-masking requires an additional Boolean operation, but since this is applied once the hash codes are already placed in the lowest levels of memory (i.e., register), it only leads to a marginal decrease in efficiency (see Section 4.6 for an empirical analysis). To derive a variational setup for hashing-based collaborative filtering, we define the likelihood of a user, u, and the likelihood of an item i, as the product over the likelihoods of the observed user specified ratings: where θ are the parameters of the (neural network) model. This formulation enforces a dual symmetric effect of users being defined by all their rated items, and items being defined by the ratings provided by all the users. To maximize the likelihood of all observed items and users, we need to maximize the likelihood of the observed ratings p θ (R u,i). Note that instead of maximizing the raw likelihood, we consider the log likelihood to derive the objective below. We assume that the likelihood of a rating, p θ (R u,i), is conditioned on two latent vectors, a user hash code z u, and an item hash code z i. To obtain the hash codes of the user and item, we assume that z u and z i each are sampled by repeating m Bernoulli trials, which have equal probability of sampling -1 and 1. This gives us the following log likelihood, which we wish to maximize: The latent vectors z u and z i are a user and item hash code, and should therefore be conditioned on the user and item respectively. To do this we first multiply and divide with the approximate posterior distributions q φ (z i |i), q ψ (z u |u): where ψ and φ are the parameters of the approximate posteriors. We can now rewrite to an expectation and apply Jensen's inequality to obtain a lower bound on the log likelihood: Since z i and z u will be sampled independently, then q φ (z i |i) and q ψ (z u |u) are independent and we can rewrite to the variational lower bound: where KL(·, ·) is the Kullback-Leibler divergence. Thus, to maximize the expected log likelihood of the observed rating, we need to maximize the conditional log likelihood of the rating, while minimising the KL divergence between the approximate posterior and prior distribution of the two latent vectors. Maximizing the expected conditional log likelihood can be considered as a reconstruction term of the model, while the KL divergence can be considered as a regularizer. Next we present the computation of the approximate posterior distributions q i φ (z i |i) and q u ψ (z u |u) (Section 3.3) and the conditional log likelihood of the rating p θ (R u,i |z u, z i) (Section 3.4). The approximate posterior distributions can be seen as two encoder functions modelled through a neural network by considering the functions as embedding layers. Each encoder function maps either a user or an item to a hash code. Next, we focus on the derivation of the encoder function for the user, as they are both computed in the same way. The probability of the j'th bit is given by: where u is the j'th entry in a learned real-valued embedding E for user u, and σ is the sigmoid function. The j'th bit is then given by: where µ (j) is either chosen stochastically by sampling µ (j) from a uniform distribution in the interval, or chosen deterministically to be 0.5, which can be used for evaluation to obtain fixed hash codes. Note that µ is sampled for each bit. As the sampling is non-differentiable, a straight-through estimator is used for backpropagation. The conditional log likelihood can be considered a reconstruction of the rating, given the user and item hash codes. We model the observed ratings as a ground truth rating with additive standard normally distributed noise, which is then discretized to the observed categorical rating. The conditional log likelihood can then be computed as: where f (z u, z i) is a function that reconstructs the rating given the user and item hash codes. Maximising the log likelihood, log p θ (R u,i |z u, z i), corresponds to minimising the mean squared error (MSE) between R u,i and f (z u, z i), which is done for training the model. Existing work on hashingbased collaborative filtering also employs a MSE objective, and thus implicitly make the same normal distribution assumption as done in this work. We define the reconstruction function to be the self-masking Hamming distance from Eq. 2: where g is a fixed affine transformation that maps the interval of the Hamming distance to the interval of the ratings, such that the minimum and maximum of the Hamming distance correspond to the minimum and maximum of the ratings. The model is now fully differentiable and can be trained end-to-end using backpropagation, such that the network is able to optimize the hash codes directly for self-masking. A depiction of the model is provided in Figure 1. It should be noted that while variational autoencoders are generative models, we do not explicitly utilize this in our model, and are primarily concerned with the reconstruction of the observed ratings. We evaluate on 4 publicly available datasets commonly used in prior work (; ; ; ; and summarized in Table 1 . Specifically, we use: two movie rating datasets, Movielens 1M 2 (ML-1M) and Movielens 10M 3 (ML-10M); a Yelp dataset with ratings of e.g., restaurant and shopping malls 4; and a book rating dataset from Amazon 5 (50% of the ratings are used for testing, 42.5% are used for training, while the last 7.5% are used for validation. We evaluate our method, VaHSM-CF, and all baselines (see Section 4.2) using Normalised Discounted Cumulative Gain (NDCG) (Järvelin & Kekäläinen, 2000), which is often used to evaluate recommender systems with non-binary ratings (or relevance values). We use the average NDCG at cutoffs {2, 6, 10} over all users and report the average for each cutoff value. We use as baselines the state-of-the-art methods for hashing-based collaborative filtering (see Section 2), standard matrix factorisation, and two different strategies for binarising the output of the matrix factorisation 6. We include the standard matrix factorization as a reference to a traditional real-valued collaborative filtering baseline, in order to highlight the performance gap between the hashing-based approaches and a real-valued approach. 7 learns user and item hash codes through a binary matrix factorization solved as a relaxed integer problem. 8 learns hash codes consisting of k blocks, where each block has r bits. A floating point weight is associated with each block for computing user-item similarities as a weighted sum of block-level Hamming distances. In the original paper, the floating point weights are not counted towards the amount of bits used, thus leading to an unfair advantage. For a fair comparison, we count each floating point weight as 16 bits in the following experimental comparison. 9 is the classical matrix factorization based collaborative filtering method, where latent real-valued vectors are learned for users and items. We set the latent dimension to be the same as the number of bits used in the hashing-based approaches. MF mean and MF median are based on MF, but use either each dimension's mean or median for doing the binary quantization to bits . We include these to highlight the large quantization loss occurring when the hash codes are not learned directly. VaH-CF is our proposed method without self-masking. We use it to show the effect of a neural variational hashing-based collaborative approach without self-masking. We train both VaHSM-CF and VaH-CF using the Adam optimizer , and tune the learning rate from the set {0.005, 0.001, 0.0005}, where 0.001 is always chosen across all data sets. The batch size is chosen from the set {100, 200, 400, 800}, where 400 was always chosen. To reduce over-fitting, Gaussian noise is added to the ratings during training, as noise injection has been found beneficial in multiple domains for variational neural models . For the noise injection, we initially set the variance of the Gaussian to 1 and reduce by a factor of 1 − 10 −4 every iteration during training. Our model is implemented using the Tensorflow Python library , and all experiments are run on Titan X GPUs. All hyper parameters for the baselines are tuned using the same set of possible values as in the original papers. For the CCCF baseline, we consider block sizes of {8, 16, 32, 64} and each floating 32 bits ML-1M ML-10M Yelp Amazon NDCG@2 NDCG@6 NDCG@10 NDCG@2 NDCG@6 NDCG@10 NDCG@2 NDCG@6 NDCG@10 NDCG@2 NDCG@6 Table 2: NDCG@K of our method (VaHSM-CF) against baselines using hash codes of length 32 and 64 bits. The missing for CCCF on Amazon are due to the model requiring more than 128 GB of RAM. Statistically significant improvements using a paired two tailed t-test at the 0.05 level, compared to the best existing hashing-based baseline (DCF), are indicated by *. point weight counts for 16 bits. We try all possible combinations that fit within the bit budget, and if a single block is chosen, then the weight is not included in the bit calculation. The are shown in Table 2, for hash code lengths of 32 and 64 bits, as these correspond to common machine word sizes. The highest NDCG per column is shown in bold (among hashingbased baselines), and statistically significantly better than the best hashing-based baseline (DCF), using a paired two tailed t-test at the 0.05 level, are indicated by an asterisk *. The Amazon for CCCF are not included, as the released implementation requires excessive amounts of RAM (>128GB) on this dataset due to the large amount of items and users. Our proposed VaHSM-CF significantly outperforms all hashing-based baselines across all datasets by up to 12%. Our VaH-CF without self-masking is second best (up to 2% better than the hashingbased baselines), although on ML-1M the VaH-CF are not statistically significantly better than DCF. This highlights the benefit of modelling hashing-based collaborative filtering with a variational deep learning based framework, which is notably different from existing hashing-based methods based on dicrete matrix factorization solved as relaxed integer problems. Most importantly however, it shows the significant improvement self-masking brings to hashing-based collaborative filtering. We observe that the top 3 baselines (including our VaH-CF) generally obtain similar scores, which highlights the difficulty of improving performance without changing how the hash codes are used (as done by self-masking). The real-valued MF baseline outperforms all the hashing-based approaches, which is to be expected as the representational power of of 32/64 floating point numbers are notably higher than 32/64 bits. However, our VaHSM-CF bridges a large part of the gap between existing hashing-based methods and MF, such that the NDCG difference in most cases is below 0.01. Additionally, the large performance decrease by both the mean or median rounding shows the large quantization error obtained if the hash codes are not learned directly, as done by our and the existing hashing-based approaches. How self-masking influences the convergence rate of the model. Figure 2a shows the convergence rate for the ML-1M dataset with and without self-masking. We see that training with self-masking significantly improves the convergence rate compared to the model without self-masking. Since the time for a single epoch is approximately the same with and without self-masking, we conclude that self-masking not only improves NDCG, but also reduces training time by a very large margin. Figure 2: 2a shows convergence using the validation NDCG@10 on ML-1M, where self-masking significantly speeds up convergence. We observe the same trend on the other datasets (see Appendix A.1). 2b Shows the test NDCG@10 when varying whether hash codes are sampled stochastically or deterministically while training and for evaluation. For example, Det. Eval + Stoc. Train corresponds to deterministic sampling of hash codes for evaluation, while sampling stochastically when training the model. Convergence plots for the remaining datasets are shown in the Appendix, where we observe the same trend. Stochastic or deterministic sampling. We investigate the effect of the sampling strategy for the hash codes (see Eq. 10) during training and evaluation. The sampling can either be deterministic (µ (j) = 0.5) or stochastic (µ (j) is sampled uniformly at random from), and does not have to be the same for training and evaluation. Figure 2b shows the performance for these 4 configurations across all datasets. We see that stochastic training with deterministic evaluation performs the best, while deterministic training and deterministic evaluation perform second best. As expected, stochastic sampling at evaluation performs significantly worse than deterministic sampling, as every item has a very small probability of being sampled such that it has a small Hamming distance to a user, even though it has a low rating (and vice versa for highly rated items). 4.6 RUNTIME ANALYSIS Self-masking has an additional cost added to the standard Hamming distance, due to the additional AND operation between the user and item hash codes (see Eq. 1 and 2). We now investigate the actual runtime cost associated with this modification. We implement both the Hamming distance and Hamming distance with self-masking efficiently in C on a machine with a 64 bit instruction set. A test environment was made with 64 bit hash codes for 100,000 users and 100,000 items. For each user, the distances were computed to all items using both the Hamming distance and Hamming distance with self-masking. We measure the actual time taken for computing the distances to all items from each user, and report the average over 50 repeated runs. All experiments are run on a single thread 10, with all users and items loaded in RAM. The code was compiled with the highest optimization level, and utilizing all optimization flags applicable to the hardware. We verified the produced assembler code used the efficient popcnt instruction. The mean experiment time was 8.0358s when using the Hamming distance, and 8.3506s when using the Hamming distance with self-masking. Thus, self-masking only adds a runtime overhead of 3.91% compared to using the standard Hamming distance. As in this setup we are only computing the distances, this can be seen as an upper bound of the actual overhead in a complete system, as the remaining operations (e.g., sorting) would be the same with and without self-masking. Thus, this provides a good trade-off compared to the large performance gains it yields. Note that the measured times are for the total 10 10 distance computations, highlighting the scalability of hashing-based methods to datasets of massive scale. For comparison, if the experiment is repeated with computing the dot product of floating point vectors of size 64, then the computation time is 648.5632s, thus close to 80x slower. We proposed an end-to-end trainable variational hashing-based collaborative filtering method, which optimizes hash codes using a novel modification to the Hamming distance, which we call selfmasking. The Hamming distance with self-masking first creates a modified item hash code, by applying an AND operation between the user and item hash codes, before computing the Hamming distance. Intuitively, this can be seen as ignoring user-specified bits when computing the Hamming distance, corresponding to applying a binary importance weight to each bit, but without using more storage and only a very marginal runtime overhead. We verified experimentally that our model outperforms state-of-the-art baselines by up to 12% in NDCG at different cutoffs, across 4 widely used datasets. These gains come at a minimal cost in recommendation time (self-masking only increased computation time by less than 4%). A.1 CONVERGENCE PLOTS Convergence plots for Yelp, Amazon, and ML-10M are shown in Figure 3. We observe a similar trend to ML-1M in Figure 2a, where the self-masking leads to a notably faster rate of convergence.' should have a representation like 'We propose a new variational hashing-based collaborative filtering approach optimized for a novel self-mask variant of the Hamming distance, which outperforms state-of-the-art by up to 12% on NDCG.' but very far from 'This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network. '.

Question: We present a simple idea that allows to record a speaker in a given language and synthesize their voice in other languages that they may not even know. These techniques open a wide range of potential applications such as cross-language communication, language learning or automatic video dubbing. We call this general problem multi-language speaker-conditioned speech synthesis and we present a simple but strong baseline for it. Our model architecture is similar to the encoder-decoder Char2Wav model or Tacotron. The main difference is that, instead of conditioning on characters or phonemes that are specific to a given language, we condition on a shared phonetic representation that is universal to all languages. This cross-language phonetic representation of text allows to synthesize speech in any language while preserving the vocal characteristics of the original speaker. Furthermore, we show that fine-tuning the weights of our model allows us to extend our to speakers outside of the training dataset. Our approach is to build a model able to generate speech in multiple languages. The model is trained 23 with multiple speakers to let the model be aware of the variations between speakers and also to 24 disentangle speech content from speaker identity. Once the model is trained, we bias the generation 25 process so that it sounds like a specific speaker. This speaker doesn't have to be in the training data. Our work builds upon recent developments in neural network based speech synthesis [Sotelo et al., 28 2017, , , languages to a universal representation. Our model is able to accomplish zero-shot accent transfer, which is very similar to zero-shot machine 42 translation, done by grounding the input from different languages to a common neural representation 43 space, followed by decoding in the audio space []. The training data consists of audio-transcript pairs. The transcript is translated into its IPA equivalent 61 before being fed to the model and the audio is transformed into an intermediate representation (e.g. WORLD vocoder parameters or spectrogram). Each speaker within the training dataset only speaks 63 a single language. However, at synthesis time, we are able to take any combination of speaker and 64 language, and produce natural sounding speech in the voice of the speaker and in the accent matching 65 that of the language. speakers. Crucially, we apply a smaller learning rate to the encoder and decoder parts of the models, and a higher one for the speaker embedding. This improved speaker fidelity considerably. After fine-tuning, the model is able to generate any text in any language 1 with the new speaker's 76 vocal identity. We conduct experiments on our models trained in two distinct settings. First, we train our model 79 with data in two languages (Bilingual Model). Second, we train our model with data in six languages (Multilingual Model). For these experiments, we used several datasets. We used an internal English dataset composed of 82 approximately 20000 speakers, with about 10 utterances per speaker. We also used the TIMIT dataset [] and DIMEx100 []. DIMEx100 is a Spanish dataset composed of 84 100 Spanish native speakers, with about 60 2-seconds utterances per speaker. For all the experiments we provide audio samples 2 rather than an exhaustive quantitative analysis. language. We show that the model is able to generate in any language for any speaker in the dataset. The model also shows robust performance on new, out-of-sample speakers after the fine-tuning step 97 (see FIG4.

We present a simple idea that allows to record a speaker in a given language and synthesize their voice in other languages that they may not even know.

Represent the TLDR for finding the corresponding scientific article!

We use representations trained without any parallel data for creating word alignments.


Question: Represent the scientific article to find a short TLDR

Question: The goal of imitation learning (IL) is to enable a learner to imitate expert behavior given expert demonstrations. Recently, generative adversarial imitation learning (GAIL) has shown significant progress on IL for complex continuous tasks. However, GAIL and its extensions require a large number of environment interactions during training. In real-world environments, the more an IL method requires the learner to interact with the environment for better imitation, the more training time it requires, and the more damage it causes to the environments and the learner itself. We believe that IL algorithms could be more applicable to real-world problems if the number of interactions could be reduced. In this paper, we propose a model-free IL algorithm for continuous control. Our algorithm is made up mainly three changes to the existing adversarial imitation learning (AIL) methods – (a) adopting off-policy actor-critic (Off-PAC) algorithm to optimize the learner policy, (b) estimating the state-action value using off-policy samples without learning reward functions, and (c) representing the stochastic policy function so that its outputs are bounded. Experimental show that our algorithm achieves competitive with GAIL while significantly reducing the environment interactions. Recent advances in reinforcement learning (RL) have achieved super-human performance on several domains BID20 BID21. On most of such domains with the success of RL, the design of reward, that explains what agent's behavior is favorable, is obvious for humans. Conversely, on domains where it is unclear how to design the reward, agents trained by RL algorithms often obtain poor policies and behave worse than what we expect them to do. Imitation learning (IL) comes in such cases. The goal of IL is to enable the learner to imitate expert behavior given the expert demonstrations without the reward signal. We are interested in IL because we desire an algorithm that can be applied to real-world problems for which it is often hard to design the reward. In addition, since it is generally hard to model a variety of real-world environments with an algorithm, and the state-action pairs in a vast majority of realworld applications such as robotics control can be naturally represented in continuous spaces, we focus on model-free IL for continuous control. A wide variety of IL methods have been proposed in the last few decades. The simplest IL method among those is behavioral cloning (BC) BID23 which learns an expert policy in a supervised fashion without environment interactions during training. BC can be the first IL option when enough demonstration is available. However, when only a limited number of demonstrations are available, BC often fails to imitate the expert behavior because of the problem which is referred to compounding error BID25 -inaccuracies compound over time and can lead the learner to encounter unseen states in the expert demonstrations. Since it is often hard to obtain a large number of demonstrations in real-world environments, BC is often not the best choice for real-world IL scenarios. Another widely used approach, which overcomes the compounding error problem, is Inverse Reinforcement Learning (IRL) BID27 BID22 BID0 BID33. Recently, BID15 have proposed generative adversarial imitation learning (GAIL) which is based on prior IRL works. Since GAIL has achieved state-of-the-art performance on a variety of continuous control tasks, the adversarial IL (AIL) framework has become a popular choice for IL BID1 BID11 BID16. It is known that the AIL methods are more sample efficient than BC in terms of the expert demonstration. However, as pointed out by BID15, the existing AIL methods have sample complexity in terms of the environment interaction. That is, even if enough demonstration is given by the expert before training the learner, the AIL methods require a large number of state-action pairs obtained through the interaction between the learner and the environment 1. The sample complexity keeps existing AIL from being employed to real-world applications for two reasons. First, the more an AIL method requires the interactions, the more training time it requires. Second, even if the expert safely demonstrated, the learner may have policies that damage the environments and the learner itself during training. Hence, the more it performs the interactions, the more it raises the possibility of getting damaged. For the real-world applications, we desire algorithms that can reduce the number of interactions while keeping the imitation capability satisfied as well as the existing AIL methods do. The following three properties of the existing AIL methods which may cause the sample complexity in terms of the environment interactions:(a) Adopting on-policy RL methods which fundamentally have sample complexity in terms of the environment interactions.(b) Alternating three optimization processes -learning reward functions, value estimation with learned reward functions, and RL to update the learner policy using the estimated value. In general, as the number of parameterized functions which are related to each other increases, the training progress may be unstable or slower, and thus more interactions may be performed during training.(c) Adopting Gaussian policy as the learner's stochastic policy, which has infinite support on a continuous action space. In common IL settings, we observe action space of the expert policy from the demonstration where the expert action can take on values within a bounded (finite) interval. As BID3 suggests, the policy which can select actions outside the bound may slow down the training progress and make the problem harder to solve, and thus more interactions may be performed during training. In this paper, we propose an IL algorithm for continuous control to improve the sample complexity of the existing AIL methods. Our algorithm is made up mainly three changes to the existing AIL methods as follows:(a) Adopting off-policy actor-critic (Off-PAC) algorithm BID5 to optimize the learner policy instead of on-policy RL algorithms. Off-policy learning is commonly known as the promising approach to improve the complexity.(b) Estimating the state-action value using off-policy samples without learning reward functions instead of using on-policy samples with the learned reward functions. Omitting the reward learning reduces functions to be optimized. It is expected to make training progress stable and faster and thus reduce the number of interactions during training.(c) Representing the stochastic policy function of which outputs are bounded instead of adopting Gaussian policy. Bounding action values may make the problem easier to solve and make the training faster, and thus reduce the number of interactions during training. Experimental show that our algorithm enables the learner to imitate the expert behavior as well as GAIL does while significantly reducing the environment interactions. Ablation experimental show that (a) adopting the off-policy scheme requires about 100 times fewer environment interactions to imitate the expert behavior than the one on-policy IL algorithms require, (b) omitting the reward learning makes the training stable and faster, and (c) bounding action values makes the training faster. We consider a Markov Decision Process (MDP) which is defined as a tuple {S, A, T, R, d 0, γ}, where S is a set of states, A is a set of possible actions agents can take, T: S×A×S → is a transition probability, R: S×A → R is a reward function, d 0: S → is a distribution over initial states, and γ ∈ is a discount factor. The agent's behavior is defined by a stochastic policy π: S×A → and Π denotes a set of the stochastic policies. We denote S E ⊂ S and A E ⊂ A as sets of states and actions observed in the expert demonstration, and S π ⊂ S and A π ⊂ A as sets of those observed in rollouts following a policy π. We will use π E, π θ, β ∈ Π to refer to the expert policy, the learner policy parameterized by θ, and a behavior policy, respectively. Given a policy π, performance measure of π is defined as J (π, R) = E ∞ t=0 γ t R(s t, a t)|d 0, T, π where s t ∈ S is a state that the agent receives at discrete time-step t, and a t ∈ A is an action taken by the agent after receiving s t. The performance measure indicates expectation of the discounted return ∞ t=0 γ t R(s t, a t) when the agent follows the policy π in the MDP. Using discounted state visitation distribution denoted by ρ π (s) = ∞ t=0 γ t P(s t = s|d 0, T, π) where P is a probability that the agent receives the state s at time-step t, the performance measure can be rewritten as J (π, R) = E s∼ρπ,a∼π R(s, a). The state-action value function for the agent following π is defined as DISPLAYFORM0 |T, π, and Q π,ν denotes its approximator parameterized by ν. We briefly describe objectives of RL, IRL, and AIL below. We refer the readers to BID15 for details. The goal of RL is to find an optimal policy that maximizes the performance measure. Given the reward function R, the objective of RL with parameterized stochastic policies π θ: S×A → is defined as follows: DISPLAYFORM0 The goal of IRL is to find a reward function based on an assumption that the discounted returns earned by the expert behavior are greater than or equal to those earned by any non-experts behavior. Technically, the objective of IRL is to find reward functions R ω: S × A → R parameterized by ω that satisfies J (π E, R ω) ≥ J (π, R ω) where π denotes the non-expert policy. The existing AIL methods adopt max-margin IRL BID0 of which objective can be defined as follows: DISPLAYFORM1 2) The objective of AIL can be defined as a composition of the objectives and as follows: DISPLAYFORM2 The objective of Off-PAC to train the learner can be described as follows: DISPLAYFORM0 The learner policy is updated by taking the gradient of the state-action value. BID5 proposed the gradient as follows: provided another formula of the gradient using "re-parameterization trick" in the case that the learner policy selects the action as a = π θ (s, z) with random variables z ∼ P z generated by a distribution P z: DISPLAYFORM1 DISPLAYFORM2 3 ALGORITHMAs mentioned in Section.1, our algorithm (a) adopts Off-PAC algorithms to train the learner policy, (b) estimates state-action value without learning the reward functions, and (c) represents the stochastic policy function so that its outputs are bounded. In this section, we first introduce (b) in 3.1 and describe (c) in 3.2, then present how to incorporate (b) and (c) into (a) in 3.3. In this subsection, we introduce a new IRL objective to learn the reward function in 3.1.1 and a new objective to learn the value function approximator in 3.1.2. Then, we show that combining those objectives derives a novel objective to learn the value function approximator without reward learning in 3.1.3. We define the parameterized reward function as R ω (s, a) = log r ω (s, a), with a function r ω: S×A → parameterized by ω. r ω (s, a) represents a probability that the state-action pairs (s, a) belong to S E × A E. In other words, r ω (s, a) explains how likely the expert executes the action a at the state s. With this reward, we can also define a Bernoulli distribution p ω: Π×S×A → such that p ω (π E |s, a) = r ω (s, a) for the expert policy π E and p ω (π|s, a) = 1 − r ω (s, a) for any other policies π ∈ Π \ {π E} which include π θ and β. A nice property of this definition of the reward is that the discounted return for a trajectory {(s 0, a 0), (s 1, a 1),...} can be written as a log likelihood with p ω (π E |s t, a t): DISPLAYFORM0 Here, we assume Markov property in terms of p ω such that p ω (π E |s t, a t) for t ≥ 1 is independent of p ω (π E |s t−u, a t−u) for u ∈ {1, ..., t}. Under this assumption, the return naturally represents how likely a trajectory is the one the expert demonstrated. The discount factor γ plays a role to make sure the return is finite as in standard RL.The IRL objective can be said to aim at assigning r ω = 1 for state-action pairs (s, a) ∈ S E × A E and r ω = 0 for (s, a) ∈ S π × A π when the same definition of the reward R ω (s, a) = log r ω (s, a) is used. Following this fashion easily leads to a problem where the return earned by the non-expert policy becomes −∞, since log r ω (s, a) = −∞ if r ω (s, a) = 0 and thus log DISPLAYFORM1 The existing AIL methods seem to mitigate this problem by trust region optimization for parameterized value function approximator BID29, and it works somehow. However, we think this problem should be got rid of in a fundamental way. We propose a different approach to evaluate state-action pairs (s, a) ∈ S π × A π. Intuitively, the learner does not know how the expert behaves in the states s ∈ S \ S E -that is, it is uncertain which actions the expert executes in the states the expert has not visited. We thereby define a new IRL objective as follows: DISPLAYFORM2 where H denotes entropy of Bernoulli distribution such that: DISPLAYFORM3 Unlike the existing AIL methods, our IRL objective is to assign p ω (π E |s, a) = p ω (π|s, a) = 0.5 for (s, a) ∈ S π × A π. This uncertainty p ω (π E |s, a) = 0.5 explicitly makes the return earned by the non-expert policy finite. On the other hand, the objective is to assign r ω = 1 for (s, a) ∈ S E × A E as do the existing AIL methods. The optimal solution for the objective satisfies the assumption of IRL: DISPLAYFORM4, even though the objective does not aim at discriminating between (s, a) ∈ S E × A E and (s, a) ∈ S π × A π, As we see in Equation FORMULA7, the discounted return can be represented as a log likelihood. Therefore, a value function approximator Q π θ following the learner policy π θ can be formed as a log probability. We introduce a function q π θ,ν: S×A → parameterized by ν to represent the approximator Q π θ,ν as follows: DISPLAYFORM0 The optimal value function following a policy π satisfies the Bellman equation Q π (s t, a t) = R(s t, a t) + γE st+1∼T,at+1∼π Q π (s t+1, a t+1). Substituting π θ for π, log r ω (s t, a t) for R(s t, a t), and log q π θ,ν (s t, a t) for Q π (s t, a t), the Bellman equation for the learner policy π θ can be written as follows: DISPLAYFORM1 We introduce additional Bernoulli distributions P ν: Π × S × A:→ and P ωνγ: Π × S × A × S × A:→ as follows: DISPLAYFORM2 Using P ν and P ωνγ, the loss to satisfy Equation FORMULA13 can be rewritten as follows: DISPLAYFORM3 We use Jensen's inequality with the concave property of logarithm in Equation FORMULA4. Now we see that the loss L(ω, ν, θ) is bounded by the log likelihood ratio between the two Bernoulli distributions P ν and P ωνγ, and L(ω, ν, θ) = 0 if P ν (π E |s t, a t) = E st+1∼T,at+1∼π θ P ωνγ (π E |s t, a t, s t+1, a t+1).In the end, learning the approximator Q π θ,ν turns out to be matching the two Bernoulli distributions. A natural way to measure the difference between two probability distributions is divergence. We choose Jensen-Shannon (JS) divergence to measure the difference because we empirically found it works better, and thereby the objective to optimize Q π θ,ν can be written as follows: DISPLAYFORM4 where D JS denotes JS divergence between two Bernoulli distributions. Suppose the optimal reward function R ω * (s, a) = log r ω * (s, a) for the objective can be obtained, the Bellman equation FORMULA13 can be rewritten as follows: DISPLAYFORM0 Recall that IRL objective aims at assigning r ω * (s t, a t) = 1 for (s t, a t) ∈ S E × A E and r ω * (s t, a t) = 0.5 for (s t, a t) ∈ S π × A π where π ∈ Π \ {π E}. Therefore, the objective FORMULA9 is rewritten as the following objective using the Bellman equation FORMULA13: DISPLAYFORM1 Thus, r ω * can be obtained by the Bellman equation FORMULA6 as long as the solution for the objective can be obtained. We optimize q π θ,ν (s t, a t) in the same way of objective as follows: DISPLAYFORM2 We use P γ ν instead of P ωνγ in objective unlike the objective. Thus, we omit reward learning that the existing AIL methods require, while learning q π θ,ν (s t, a t) to obtain r ω *. Initialize time-step t = 0 and receive initial state s0 5:while not terminate condition do 6:Execute an action at = π θ (st, z) with z ∼ Pz and observe new state st+1 7:Store a state-action triplet (st, at, st+1) in B β. 8: DISPLAYFORM3 end while 10:for u = 1, t do 11:Sample mini-batches of triplets (s t ′ +1). end for 16: end for Recall that the aim of IL is to imitate the expert behavior. It can be summarized that IL attempts to obtain a generative model the expert has over A conditioned on states in S. We see that the aim itself is equivalent to that of conditional generative adversarial networks (cGANs) BID19. The generator of cGANs can generate stochastic outputs of which range are bounded. As mentioned in Section 1, bounding action values is expected to make the problem easier to solve and make the training faster. In the end, we adopt the form of the conditional generator to represent the stochastic learner policy π θ (s, z). The typical Gaussian policy and the proposed policy representations with neural networks are described in FIG0 Algorithm.1 shows the overview of our off-policy actor-critic imitation learning algorithm. To learn the value function approximator Q π θ,ν, we adopt a behavior policy β as π in the second term in objective We employ a mixture of the past learner policies as β and a replay buffer B β BID20 ) to perform sampling s t ∼ ρ π, a t ∼ π and s t+1 ∼ T. The buffer B β is a finite cache and stores the (s t, a t, s t+1) triplets in a first-in-first-out manner while the learner interacts with the environment. The approximator Q π θ,ν (s t, a t) = log q π θ,ν (s t, a t) takes (−∞, 0]. With the approximator, using the gradient to update the learner policy always punish (or ignore) the learner's actions. Instead, we adopt the gradient which directly uses Jacobian of Q π θ,ν.As do off-policy RL methods such as BID20 and, we use the target value function approximator, of which parameters are updated to track ν, to optimize Q π θ,ν. We update Q π θ,ν and π θ at the end of each episode rather than following each step of interaction. In recent years, the connection between generative adversarial networks (GAN) BID10 and IL has been pointed out BID15 BID6. BID15 show that IRL is a dual problem of RL which can be deemed as a problem to match the learner's occupancy measure BID31 to that of the expert, and that a choice of regularizer for the cost function yields an objective which is analogous to that of GAN. Their algorithm, namely GAIL, has become a popular choice for IL and some extensions of GAIL have been proposed BID1 BID11 BID16. However, those extensions have never addressed reducing the number of interactions during training. There has been a few attempts that try to improve the sample complexity in IL literatures, such as Guided Cost Learning (GCL) BID7. However, those methods have worse imitation capability in comparison with GAIL, as reported by BID8. As detailed in section 5, our algorithm have comparable imitation capability to GAIL while improving the sample complexity. Hester & Osband FORMULA7 proposed an off-policy algorithm using the expert demonstration. They address problems where both demonstration and hand-crafted rewards are given. Whereas, we address problems where only the expert demonstration is given. There is another line of IL works where the learner can ask the expert which actions should be taken during training, such as DAgger BID24, SEARN (Daumé &), SMILe BID25, and AggreVaTe BID26. As opposed to those methods, we do not suppose that the learner can query the expert during training. In our experiments, we aim to answer the following three questions: Q1. Can our algorithm enable the learner to imitate the expert behavior? Q2. Is our algorithm more sample efficient than BC in terms of the expert demonstration? Q3. Is our algorithm more efficient than GAIL in terms of the training time? To answer the questions above, we use five physics-based control tasks that are simulated with MuJoCo physics simulator BID32. See Appendix A for the description of each task. In the experiments, we compare the performance of our algorithm, BC, GAIL, and GAIL initialized by BC 23. The implementation details can be found in Appendix B. We train an agent on each task by TRPO BID28 using the rewards defined in the OpenAI Gym BID2, then we use the ing agent with a stochastic policy as the expert for the IL algorithms. We store (s t, a t, s t+1) triplets during the expert demonstration, then the triplets are used as training samples in the IL algorithms. In order to study the sample efficiency of the IL algorithms, we arrange two setups. The first is sparse sampling setup, where we randomly sample 100 (s t, a t, s t+1) triplets from each trajectory which contains 1000 triplets. Then we perform the IL algorithms using datasets that consist of several 100s triplets. Another setup is dense sampling setup, where we use full (s t, a t, s t+1) triplets in each trajectory, then train the learner using datasets that consist of several trajectories. If an IL algorithm succeeds to imitate the expert behavior in the dense sampling setup whereas it fails in the sparse sampling setup, we evaluate the algorithm as sample inefficient in terms of the expert demonstration. The performance of the experts and the learners are measured by cumulative reward they earned in a trajectory. We run three experiments on each task, and measure the performance during training. Figure 2 shows the experimental in both sparse and dense sampling setup. In comparison with GAIL, our algorithm marks worse performance on Walker2d-v1 and Humanoid-v1 with the datasets of the smallest size in sparse sampling setup, better performance on Ant-v1 in both setups, and competitive performance on the other tasks in both setups. Overall, we conclude that our algorithm is competitive with GAIL with regards to performance. That is, our algorithm enables the learner to imitate the expert behavior as well as GAIL does. BC imitates the expert behavior successfully on all tasks in the dense sampling setup. However, BC often fails to imitate the expert behavior in the sparse sampling setup with smaller datasets. Our algorithm achieves better performance than BC does all over the tasks. It shows that our algorithm is more sample efficient than BC in terms of the expert demonstration. Figure 3 shows the performance plot curves over validation rollouts during training in the sparse sampling setup. The curves on the top row in Figure 3 show that our algorithm denoted by Ours trains the learner more efficiently than GAIL does in terms of training time. In addition, the curves on the bottom row in Figure 3 show that our algorithm trains the learner much more efficiently than GAIL does in terms of the environment interaction. As opposed to BID15 suggestion, GAIL initialized by BC (BC+GAIL) does not improve the sample efficiency, but rather harms the leaner's performance significantly. We conducted additional ablation experiments to demonstrate that our proposed method described in Section.3 improves the sample efficiency. FIG2 shows the ablation experimental on Antv1 task. Ours+OnP, which denotes an on-policy variant of Ours, requires 100 times more interactions than Ours. The with Ours+OnP suggests that adopting off-policy learning scheme instead of on-policy one significantly improves the sample efficiency. Ours+IRL(D) and Ours+IRL(E) are variants of Ours that learn value function approximators using the learned reward function with the objective and FORMULA9, respectively. The with Ours+IRL(D) and Ours+IRL(E) suggests that omitting the reward learning described in 3.1 makes the training stable and faster. The with Ours+GP, which denotes a variant of Ours that adopts the Gaussian policy, suggests that bounding action values described in 3.2 makes the training faster and stable. The with Ours+DP, which denotes a variant of Ours that has a deterministic policy with fixed input noises, fails to imitate the expert behavior. It shows that the input noise variable z in our algorithm plays a role to obtain stochastic policies. In this paper, we proposed a model-free IL algorithm for continuous control. Experimental showed that our algorithm achieves competitive performance with GAIL while significantly reducing the environment interactions. A DETAILED DESCRIPTION OF EXPERIMENT TAB0 summarizes the description of each task, the performance of an agent with random policy, and the performance of the experts. We implement our algorithm using two neural networks with two hidden layers. Each network represents π θ and q ν. For convenience, we call those networks for π θ and q ω as policy network (PN) and Q-network (QN), respectively. PN has 100 hidden units in each hidden layer, and its final output is followed by hyperbolic tangent nonlinearity to bound its action range. QN has 500 hidden units in each hidden layer and a single output is followed by sigmoid nonlinearity to bound the output between. All hidden layers are followed by leaky rectified nonlinearity BID18. The parameters in all layers are initialized by Xavier initialization BID9. The input of PN is given by concatenated vector representations for the state s and noise z. The noise vector, of which dimensionality corresponds to that of the state vector, generated by zero-mean normal distribution so that z ∼ P z = N. The input of QN is given by concatenated vector representations for the state s and action a. We employ RMSProp BID14 for learning parameters with a decay rate 0.995 and epsilon 10 −8. The learning rates are initially set to 10 −3 for QN and 10 −4 for PN, respectively. The target QN with parameters ν ′ are updated so that ν ′ = 10 −3 * ν+(1−10 −3) * ν ′ at each update of ν. We linearly decrease the learning rates as the training proceeds. We set minibatch size of (s t, a t, s t+1) triplets 64, the replay buffer size |B β | = 15000, and the discount factor γ = 0.85. We sample 128 noise vectors for calculating empirical expectation E z∼Pz of the gradient. We use publicly available code (https://github.com/openai/imitation) for the implementation of GAIL and BC. Note that, the number of hidden units in PN is the same as that of networks for GAIL. All experiments are run on a PC with a 3.30 GHz Intel Core i7-5820k Processor, a GeForce GTX Titan GPU, and 32GB of RAM.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

In this paper, we proposed a model-free, off-policy IL algorithm for continuous control. Experimental results showed that our algorithm achieves competitive results with GAIL while significantly reducing the environment interactions.

Represent the short summary for finding the corresponding scientific article (long)

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.