Question: Represent the scientific article to find a short TLDR

Question: Although word analogy problems have become a standard tool for evaluating word vectors, little is known about why word vectors are so good at solving these problems. In this paper, I attempt to further our understanding of the subject, by developing a simple, but highly accurate generative approach to solve the word analogy problem for the case when all terms involved in the problem are nouns. My demonstrate the ambiguities associated with learning the relationship between a word pair, and the role of the training dataset in determining the relationship which gets most highlighted. Furthermore, my show that the ability of a model to accurately solve the word analogy problem may not be indicative of a model’s ability to learn the relationship between a word pair the way a human does. Word vectors constructed using Word2vec BID6, BID8 ) and Glove BID9 ) are central to the success of several state of the art models in natural language processing BID1, BID2, BID7, BID11 ). These vectors are low dimensional vector representations of words that accurately capture the semantic and syntactic information about the word in a document. The ability of these vectors to encode language is best illustrated by their efficiency at solving word analogy problems. The problem involves predicting a word, D, which completes analogies of the form'A:B:: C:D'. For example, if the phrase is''King:Queen:: Man:D', then the appropriate value of D is Woman. Word2vec solves these problems by observing that the word vectors for A, B, C and D satisfy the equation V ec(D) ≈ V ec(C) + V ec(B) − V ec(A) in several cases. Although this equation accurately resolves the word analogy for a wide variety of semantic and syntactic problems, the precise dynamics underlying this equation are largely unknown. Part of the difficulty in understanding the dynamics is that word vectors are essentially'black boxes' which lack interpretability. This difficulty has been overcome in large part due to the systematic analyses of Levy, Goldberg and colleagues, who have derived connections between word vectors and the more human-interpretable count based approach of representing words. They show that 1) there are mathematical equivalences between Word2vec and the count based approach BID4 ), 2) that the count based approach can produce comparable to Word2vec on word analogy problems BID3 ) and more generally, 3) that the count based approach can perform as well as Word2vec on most NLP tasks when the hyper-parameters in the model are properly tuned BID5. Their (see section 9 in BID3) demonstrate that V ec(B) − V ec(A) is likely capturing the'common information' between A and B, and this information is somehow being'transferred' on to C to compute D.Still the question remains, how is this transference process taking place? The answer would provide insight into the topology of word vectors and would help us to identify gaps in our understanding of word vectors. In this paper, I attempt to gain insights into the transference process by building a simple generative algorithm for solving semantic word analogy problems in the case where A, B, C and D are nouns. My algorithm works in two steps: In the first step, I compute a list of nouns that likely represent the information that is common to both A and B. In the second step, I impose the information about the nouns obtained in the first step on to C to compute D. Both steps of the algorithm work only on word counts; therefore, it is possible to precisely understand how and why D is generated in every word analogy question. Despite the simplicity of my approach, the algorithm is able to produce comparable to Word2vec on the semantic word analogy questions, even using a very small dataset. My study reveals insights into why word vectors solve certain classes of word analogy problems much better than others. I show that there is no universal interpretation of the information contained in V ec(B) − V ec(A) because the'common information' between A and B is strongly dependent on the training dataset. My reveal that a machine may not be'learning' the relationship between a pair of words the way a human does, even when it accurately solves an analogy problem. Problem Statement. In this paper, I analyze a variant of the semantic word analogy problem studied in BID8. The problem can be stated as follows: given 3 nouns, A, B and C, appearing in the text, T, find a fourth noun D such that the semantic relationship (R) between A and B is the same as the semantic relationship between C and D. Here, R describes an'is a' relationship; for instance, if A = Beijing and B = China, then R is likely to be capital since Beijing is a capital of China. Typically, the relationship between A and B will not be unique; in the example above, we could also have said Beijing is a city in China, or Beijing is a center of tourism in China. The dataset: For my analysis, the text, T, comprises the first billion characters from Wikipedia. This dataset contains less than 10 % of the information present in Wikipedia. The data can be downloaded from http://mattmahoney.net/dc/enwik9.zip and pre-processed using wikiextractor detailed in http://medialab.di.unipi.it/wiki/Wikipedia_Extractor. The raw data is divided into several unrelated chapters for e.g., there is a chapter on'Geography of Angola','Mouthwash Antiseptic' etc. As part of the pre-processing I remove all those chapters containing less than 5 words. Analysis questions: I test the efficacy of my proposed solution using a subset of the semantic word analogy problems compiled by BID8 that is relevant to this study. The test set used here comprise 8,363 problems 1 divided into 4 categories: common capitals (e.g., Athens:Greece::Oslo:Norway), all capitals (e.g., Vienna:Austria::Bangkok:Thailand), currencies (e.g., Argentina:peso::Hungary: forint) and cities in states of the U.S (e.g., Dallas:Texas::Henderson:Nevada). The original dataset comprises questions in a fifth category of gender inflections (e.g., grandfather:grandmother::father:mother) which is left out of the analysis because many of the problems involve pronouns (e.g., his:her::man:woman).Word2vec. I compare the produced by my method with those obtained using Word2vec. Word2vec derives low dimensional vector representations for words such that'similar' words occupy'similar' regions in a vector space. Given a sequence of words w 1, w 2... w T, Word2vec maximizes the log probability DISPLAYFORM0 where w is the window size, and p(w t+j |w t) is a function of the word vectors for w t+j and w t respectively; for detailed insights into the functioning of Word2vec see BID0 and. Given the word vectors V ec(A), V ec(B) and V ec(C) for A, B and C respectively, Word2vec derives D as the word whose word vector best satisfies the relationship DISPLAYFORM1 The critical implicit assumption made in Equation FORMULA0 is that the w words surrounding w i on either side, namely [w i−w, w i−w+1, . . . w i−1, w i+1 . . . w i+w−1, w i+w] contains the most semantically and syntactically relevant words to w i. In the next section, I will explain how a slight generalization of this forms the basis for my algorithm. The goal is to solve the word analogy problem using a simple, generative window based approach. I begin my analysis by noting that all terms relevant in the analysis (A, B, C, D, and R) are nouns. Accordingly, I construct a new document, T, comprising only the nouns appearing in T, stored in the order in which they appear 2 For convenience of notation, I will assume that there are H nouns in T, and the i th noun appearing in T is represented as T i (i.e., T [i] = T i ). Since the same noun likely appear multiple times in the text, I use the set Q(X) = {i|T i = X} to indicate the locations of the noun, X in T.The key idea in Word2vec is that the words surrounding a given word in the text contain rich semantic information about that word. More generally, we expect the nouns surrounding a given noun to contain rich semantic information about the noun. This implies that for certain values of w, the context of T i defined as DISPLAYFORM0 will contain nouns that are semantically related to DISPLAYFORM1 is a metric distance function describing the number of nouns separating y and T i in T, and w is the window size. Clearly, equation FORMULA2 is likely to hold for small values of w, and not likely to hold for very large values of w. Accordingly, I make the following assumption: Assumption 1. There exists a maximal window size, w * = 2s, such that the nouns present in DISPLAYFORM2 The thus far describe contexts around one noun. We are interested in nouns that are semantically related to 2 nouns. Therefore, I definẽ DISPLAYFORM3 which describes the combined context of F (T i, w) and F (T j, w), when F (T i, w) and F (T j, w) overlap i.e., when i < j ≤ i + w. For any 2 nouns, A and B, and the set, S(A, B, w) defined as DISPLAYFORM4 we have the following : Proposition 1. If assumption 1 holds true, then all the nouns present in W ∈ S(A, B, s) will be semantically related to both A and B.Proof. For every noun N ∈ W we have DISPLAYFORM5 i.e., N belongs to one of the contexts of B and is therefore, semantically related to B. Similarly, it can be shown that N is semantically related to A.Proposition 1 describes the ideal scenario; realistically, we do not expect assumption 1 to hold exactly and therefore, we expect W ∈ S(A, B, s) to contain a higher frequency of nouns that are relevant to both A and B, and a lower frequency of nouns that are relevant to only either A or B. In particular, the higher the frequency of the noun appearing in the list Set L C to empty List 3: DISPLAYFORM6 DISPLAYFORM7 if F (T i, s) overlaps with F (T j, s) and j > i then DISPLAYFORM8 Equation FORMULA5 6: DISPLAYFORM9 Once N AB is computed using algorithm 1, we can derive D from C as the most frequently appearing noun in the list for every X in N AB do 4: DISPLAYFORM10 DISPLAYFORM11 if F (T i, s) overlaps with F (T j, s) and j > i then DISPLAYFORM12 return MostCommon(L D, 1) D is the most frequent noun in L D Algorithms 1 and 2 described above have two hyper-parameters: s and k. A grid search on the parameter values suggests that the improvement of the approach begins to saturate around s = 10 and k = 20; these are the parameter values used in the remainder of the analysis unless specified otherwise. TAB0 shows that the approach described in this paper does better than Word2vec on 3 out of 4 categories when the word vectors are trained using the same dataset. The current approach beats the obtained by Word2vec in 2 out of 4 categories even when the word vectors are trained on the entire Wikipedia 2014 corpus, which has more than 10 times the amount of data used in the current analysis. Algorithm 1 assumes that the k nouns in N AB are equally likely candidates for the relationship between A and B. While this assumption is a good starting point, it's not precise enough. Table 2 shows that the more frequently co-occurring nouns with A and B capture more information about the relationship between A and B than the less frequently co-occurring nouns. This suggests that 1) word vectors are likely capturing information about the nouns co-occurring with A and B, weighted by their frequency of occurrence with A and B, and 2) that the most frequently co-occurring noun with A and B likely represents the Maximum Likelihood Estimate (MLE) of the relationship between A and B. TAB2 shows the 5 most frequently observed MLE values for questions in each of the 4 categories, listed by their frequency of occurrence. Although most of the MLE values in the table make intuitive sense, some (particularly the MLEs for 'common capitals') do not; I will revisit this point later. The remainder of my analysis proceeds in two parts: In the first step, I discuss some of the problems associated with estimating word vectors for infrequently occurring words in the dataset and in the second step, I describe problems one might encounter with word vectors corresponding to frequently occurring words in the dataset. There is substantial variation in the prediction ability between the categories; both the current approach and Word2vec perform worse predicting currencies than the other 3 categories. This is probably because currencies appear far less frequently in the training dataset as compared to nouns in the other categories as demonstrated in Table 4. The lack of training data likely in poor estimates of the relationship between A and B and accordingly, poor estimates of D. Similar problems will likely by observed with word analogy problems involving words appearing less frequently than the currencies in the current dataset. Increasing the size of the dataset will resolve some of these issues relating to data scarcity, but to what extent? Will word vectors corresponding to most of the words trained on a larger dataset be accurate? To answer this question, consider the word vectors obtained by training Word2vec on the entire Wikipedia corpus, which comprises approximately 1.5 billion tokens, of which approximately 5.8 million are unique. The most popular token is the which appears 86 million times. 3. From Zipf's law, we expect the frequency of occurrence of a particular word in the text to be inversely proportional to its frequency rank. Therefore, a word with a frequency rank of 1 million will appear approximately 86 times. Since Word2vec is fitting a 200 dimensional vector corresponding to this word using data from 86 points, I expect the estimate of the vector to be unreliable. This suggests that word vectors corresponding to at least 80 % of the unique words trained on the entire Wikipedia corpus will be unreliable. In general, any dataset will contain some percentage of low frequency words for which accurate word vectors cannot be adequately estimated. Care should be taken to filter these out as is done in BID9.3 http://imonad.com/seo/wikipedia-word-frequency-list/ Table 2: Frequency of co-occurence matters. Accuracy of the current approach at solving the word analogy problem for different values of k. Increasing the value of k produces diminishing returns beyond k = 5; the improvements on going from k = 10 to k = 20 are nominal. In the'common capitals' category, the countries and capitals being considered appear with high frequencies (see table 4), and are therefore not plagued by the data scarcity issues described in the previous section. Furthermore, the algorithm described in this paper is able to solve the word analogy problem in this category with a high accuracy as shown in table 1. These facts together seem to imply that the approach is learning the relationship capital accurately from the data. However, as shown in table 3, capital is not even in the top 5 most likely candidates for the relationship between A and B. This suggests that a model may not be'learning' the relationship between a pair of words the way a human does, even when it accurately solves the word analogy problem. DISPLAYFORM0 To further elaborate on this point, consider figure 2 of BID8, wherein the authors demonstrate that the projection of the vector describing the relationship between a country and its capital has nearly the same orientation for ten (country,capital) pairs. The authors attribute this fixed orientation to the ability of word vectors to learn the'is capital' relationship without any supervised effort. Table 5 indicates that word vectors might actually be learning the relationship from information about wars afflicting cities in different countries. 4 Since the attacks in a war are generally targeted at the capital city, word vectors appear to be learning this relationship simply as a of correlations. These show that the context that a model uses to infer the relationship between a pair words may not match the context that humans typically use to infer the relationship between the same pair of words. The ambiguity in context can lead to some paradoxical properties of word relationships (see table 6) such as:• Pluralization can change the relationship between words. The relationship for the pair Bear:Lion derives from the fact that they are both animals, but the relationship for the pair Bears:Lions derives from the fact that they are both names of sports teams. A corollary of this is that the of word analogy questions need not be symmetric; for the word analogy question Bears:Lions::Bear:D the context is'sports game', but for the word analogy question Bear:Lion::Bears:D the context is'animal'. 4 Most of the relationships seem to derive from World War II Table 4: Median counts Median number of times A and B appear in the dataset for each of the categories. The median counts for the frequencies are far less than that for any other group. Table 5: Derived relationship between countries and their capitals. The five most commonly cooccurring nouns with countries and their capitals. The countries chosen in this list are the same as those considered in figure 2 of BID8. The model appears to be learning information about the wars that affected the capital city of these countries.. Here, there is ambiguity in the relationship despite the fact that in both cases, the nouns, Bears and Lions are being used in the context of sports teams. The relationship which gets over-emphasized in the word vector is strictly a function of the data being used to train the word vectors.• Inferences drawn from word analogies may produce counter-intuitive . The relationships for the Lions:Giants and Dolphins:Giants pairs are deriving from the fact that the Lions, the Dolphins, and the Giants are teams in the National Football League. However, the relationship for the Lions:Dolphins pair is deriving from the fact that they are both animals. The generative approach described above makes a critical assumption that is not required by Word2vec -that the answer to the word analogy problem being posed will always be a noun. Indeed, Word2vec produces high accuracies even on questions where the answers to the word analogy questions are not nouns, by'learning' the part of speech required by the outcome. I believe that this learning takes place because word vectors corresponding to words having the same Part of Speech (POS) tag lie in the same subspace of the word vector space. The word analogy problem relies on the fact that DISPLAYFORM0 5 https://en.wikipedia.org/wiki/Bears%E2%80%93Lions_rivalry 6 https://en.wikipedia.org/wiki/Brisbane_Lions Therefore, if a POS subspace did exist, V ec(D) would belong to Span(V ec(A), V ec(B), V ec(C)), and D would be forced to have the same POS tag as either A or B or C. For the word analogy questions considered in the previous section, A, B and C are all nouns, and therefore I would expect D to also be a noun, as is the case. The idea behind equation FORMULA16 is that V ec(B) − V ec(A) is capturing information that is common to both A and B, and the information that is common to both A and B is also common to C and D. If a POS subspace does exist, then for any 2 nouns, A and B, the'common information' (described by V ec(B) − V ec(A)) will lie in the noun subspace of the vector space, and can be expressed as a linear combination of the information in other nouns in the text; this is the possibly why algorithm 1 is able to express the common information between 2 nouns solely in terms of other nouns. The question remains, what does'common information' mean? Is Canada Air the common information between Canada and Air? Our intuitive guess would be no, as is the claim made in BID8, since there is nothing'airline' about Canada. To test their claim, I construct algorithm 3 to derive the most likely interpretation of a two-word phrase'A B'. Algorithm 3 is a minor variant of algorithm 1, which finds all nouns co-occurring with A and B under the stricter constraint that B is always the noun succeeding A. If the claim in BID8 is true, we would expect the information captured by algorithm 1 to be drastically different from that captured by algorithm 3. Equation FORMULA5 6:L C ← L C + [w for w inF (i, j, s) if (w = A and w = B)] Equation FORMULA8 7:return MostCommon(L C, k) N AB contains the k most frequent nouns in L C This is not what happens. TAB5 shows that both, algorithm 1 and algorithm 3 capture information about Canada Air being an airline equally well. But, when considering the common information between the words Pacific and Southwest, algorithm 3 captures information about the airline whereas algorithm 1 captures information about the region. Similar contradictions are observed when considering names of famous people -for both algorithms, the common information between Larry and Page is Google. But, with the words Stephen and King, algorithm 3 captures information about the novelist while algorithm 1 captures information about history. These suggest that the notion of'common information' is very subjective, and is strongly influenced by the data used to train the word vectors. Although problems have become a mainstay in illustrating the efficacy of word vectors, little is known about the dynamics underlying the solution. In this paper, I attempt to improve our understanding by providing a simple generative approach to solve the problem for the case when A, B, C and D are nouns. My approach proceeds by first estimating the relationship between the (A, B) pair, and then transferring this relationship to C to compute D. My demonstrates the high ambiguity associated with estimating the relationship between a word pair, and the role of the training dataset in determining the relationship which gets most represented. My analysis shows that even when the model predicts D accurately, it is difficult to infer the relationship the model learns about the (A, B) pair.

represent the tldr for finding the corresponding scientific article
Example:
Provided: "We propose a new attack for taking full control of neural policies in realistic settings." Match: "We propose a new perspective on adversarial attacks against deep reinforcement learning agents. Our main contribution is CopyCAT, a targeted attack able to consistently lure an agent into following an outsider's policy. It is pre-computed, therefore fast inferred, and could thus be usable in a real-time scenario. We show its effectiveness on Atari 2600 games in the novel read-only setting. In the latter, the adversary cannot directly modify the agent's state -its representation of the environment- but can only attack the agent's observation -its perception of the environment. Directly modifying the agent's state would require a write-access to the agent's inner workings and we argue that this assumption is too strong in realistic settings. We are interested in the problem of attacking sequential control systems that use deep neural policies. In the context of supervised learning, previous work developed methods to attack neural classifiers by crafting so-called adversarial examples. These are malicious inputs particularly successful at fooling deep networks with high-dimensional input-data like images. Within the framework of sequential-decision-making, previous works used these adversarial examples only to break neural policies. Yet the attacks they build are rarely applicable in a real-time setting as they require to craft a new adversarial input at each time step. Besides, these methods use the strong assumption of having a write-access to what we call the agent's inner state -the actual input of the neural policy built by the algorithm from the observations-. When taking this assumption, the adversary -the algorithm attacking the agent-is not placed at the interface between the agent and the environment where the system is the most vulnerable. We wish to design an attack with a more general purpose than just shattering a neural policy as well as working in a more realistic setting. Our main contribution is CopyCAT, an algorithm for taking full-control of neural policies. It produces a simple attack that is: targeted towards a policy, i.e., it aims at matching a neural policy's behavior with the one of an arbitrary policy; only altering observation of the environment rather than complete agent's inner state; composed of a finite set of pre-computed state-independent masks. This way it requires no additional time at inference hence it could be usable in a real-time setting. We introduce CopyCAT in the white-box scenario, with read-only access to the weights and the architecture of the neural policy. This is a realistic setting as prior work showed that after training substitute models, one could transfer an attack computed on these to the inaccessible attacked model . The context is the following: We are given any agent using a neuralnetwork for decision-making (e.g., the Q-network for value-based agents, the policy network for actor-critic or imitation learning methods) and a target policy we want the agent to follow. The only thing one can alter is the observation the agent receives from the environment and not the full input of the neural controller (the inner state). In other words, we are granted a read-only access to the agent's inner workings. In the case of Atari 2600 games, the agents builds its inner state by stacking the last four observations. Attacking the agent's inner state means writing in the agent's memory of the last observations. The computed attack should be inferred fast enough to be used in real-time. We stress the fact that targeting a policy is a more general scheme than untargeted attacks where the goal is to stop the agent from taking its preferred action (hoping for it to take the worst). It is also more general than the targeted scheme of previous works where one wants the agent to take its least preferred action or to reach a specific state. In our setting, one can either hard-code or train a target policy. This policy could be minimizing the agent's true reward but also maximizing the reward for another task. For instance, this could mean taking full control of an autonomous vehicle, possibly bringing it to any place of your choice. We exemplify this approach on the classical benchmark of Atari 2600 games. We show that taking control of a trained deep RL agent so that its behavior matches a desired policy can be done with this very simple attack. We believe such an attack reveals the vulnerability of autonomous agents. As one could lure them into following catastrophic behaviors, autonomous cars, robots or any agent with high dimensional inputs are exposed to such manipulation. This suggests that it would be worth studying new defense mechanisms that could be specific to RL agents, but this is out of the scope of this paper. In Reinforcement Learning (RL), an agent interacts sequentially with a dynamic environment so as to learn an optimal control. To do so, the problem is modeled as a Markov Decision Process. It is a tuple {S, A, P, r, γ} with S the state space, A the action space we consider as finite in the present work, P the transition kernel defining the dynamics of the environment, r a bounded reward function and γ ∈ a discount factor. The policy π maps states to distributions over actions: π(·|s). The (random) discounted return is defined as G = t≥0 γ t r t. The policy π is trained to maximize the agent expected discounted return. The function V π (s) = E π [G|s 0 = s] denotes the value function of policy π (where E π [·] denotes the expectation over all possible trajectories generated by policy π). We also call µ 0 the initial state distribution and ρ(π) = E s∼µ0 [V π (s)] the expected cumulative reward starting from µ 0. Value-based algorithms use the value function, or more frequently the action-value function Q π (s, a) = E π [G|s 0 = s, a 0 = a], to compute π. To handle large state spaces, deep RL uses deep neural networks for function approximation. For instance, value-based deep RL parametrize the action-value function Q ω with a neural network of parameters ω and deep actor-critics directly parametrize their policy π θ with a neural network of parameters θ. In both cases, the taken action is inferred by a forward-pass in a neural network. Adversarial examples were introduced in the context of supervised classification. Given a classifier C, an input x, a bound on a norm., an adversarial example is an input x = x + η such that C(x) = C(x) while x − x ≤. Fast Gradient Sign Method (FGSM) is a widespread method for generating adversarial examples for the L ∞ norm. From a linear approximation of C, it computes the attack η as: with l(θ, x, y) the loss of the classifier and y the true label. As an adversary, one wishes to maximize the loss l(θ, x + η, y) w.r.t. η. Presented this way, it is an untargeted attack. It pushes C towards misclassifying x in any other label than y. It can easily be turned into a targeted attack by, instead of l(θ, x + η, y), optimizing for −l(θ, x + η, y target) with y target the label the adversary wants C to predict for x. This attack, optimized for the L ∞ norm can also be turned into an L 2 attack by taking: As shown by Eq. equation 1 and Eq. equation 2, these attacks are computed with one single step of gradient, hence the term "fast". These two attacks can be turned into -more efficient, yet sloweriterative methods by taking several successive steps of gradients. These methods will be referred to as iterative-FGSM. When using deep networks to compute its policy, an RL agent can be fooled the same way as a supervised classifier. As a policy can be seen as a mapping S → A, untargeted FGSM can be applied to a deep RL agent to stop it from taking its preferred action: a * = arg max a∈A π(a|s). Similarly targeted FGSM can be used to lure the agent into taking a specific action. Yet, this would mean having to compute a new attack at each time step, which is generally not feasible in a real-time setting. Moreover, with this formulation, it needs to directly modify the agent's inner state, the input of the neural policy, which is a strong assumption. In this work, we propose CopyCAT. It is an attack whose goal is to lure an agent into having a given behavior, the latter being specified by another policy. CopyCAT's goal is not only to lure the agent into taking specific actions but to fully control its behavior. Formally, CopyCAT is composed of a set of additive masks ∆ = {δ i} 1≤i≤|A| than can be used to drive a policy π to follow any policy π target. Each additive mask δ i is pre-computed to lure π into taking a specific action a i when added to the current observation regardless of the content of the observation. It is, in this sense, a universal attack. CopyCAT is an attack on raw observations and, as ∆ is pre-computed, it can be used online in a real-time setting with no additional computation. Notations We denote π the attacked policy and π target the target policy. At time step t, the policy π outputs an action a t taking the state s t as input. The agent state is internally computed from the past observations and we denote f the observations-to-state function: Data Collection In order to be pre-computed, CopyCAT needs to gather data from the agent. By watching the agent interacting with the environment, CopyCAT gathers a dataset D of K episodes made of observations: We recall that the objective in this setting is for CopyCAT to work with a read-only access to the inner workings of the agent. We thus stress that D is made of observations rather than states. If CopyCAT is successful, π is going to behave as π target and thus may experience observations out of the distribution represented in D. Yet, as will be shown, CopyCAT transfers to unseen observations. We hypothesize that, as we build a universal attack, the learned attack is able to move the whole support of observations in a region of R N where π chooses a precise action. Training A natural strategy for building an adversarial example targeted towards labelŷ is the following. Given a classifier P(y|x) parametrized with a neural network and an input example x, one computes the adversarial examplex = x + δ by maximizing log P(ŷ|x) subject to the constraint: The adversary then performs either one step of gradient (FGSM) or uses an iterative method to solve the optimization problem. Instead, CopyCAT is built for its masks to be working whatever the observation it is applied to. For each a i ∈ A we build δ i, the additional masks luring π into taking action a i, by maximizing over δ i: We restrict the method to the case where f, the function building the agent's inner state from the observations, is differentiable. Eq. 3 is optimized by alternating between gradient steps with adaptive learning rate and projection steps onto the L ∞ -ball of radius. Unlike FGSM, CopyCAT is a full optimization method. It does not take one single step of gradient. CopyCAT has two main parameters: ∈ R +, a hard constraint on the L ∞ norm of the attack and α ∈ R +, a regularization parameter on the L 2 norm of the attack. Inference Once ∆ is computed, the attack can be used on π to make it follow any policy π target. At each time step t and given past observations, π target infers an action a target t given the sequence of observations and the corresponding mask δ a target t ∈ ∆ is applied to the last observation o t before being passed to the agent. Vulnerabilities of neural classifiers were highlighted by and several methods were developed to create the so-called adversarial examples, maliciously crafted inputs fooling deep networks. In sequential-decision-making, previous works use them to attack deep reinforcement learning agents. However these attacks are not always realistic. The method from uses fast-gradient-sign method , for the sole purpose of destroying the agent's performance. What's more, it has to craft a new attack at each time step. This implies backpropagating through the agent's network, which is not feasible in real-time. Moreover, it modifies directly the inner state of the agent by writing in its memory, which is a strong assumption to take on what component of the agent can be altered. The approach of allows the number of attacked states to be divided by four, yet it uses the heavy optimization scheme from for crafting their adversarial examples. This is, in general, not doable in a real-time setting. They also take the same strong assumption of having a read & write-access to the agent's inner workings. To the best of our knowledge, they are the first to introduce a targeted attack. However, the setting is restricted to targeting one dangerous state. proposes a method to lure the agent into taking its least preferred action in order to reduce its performance but still uses computationally heavy iterative methods at each time step. proposed an adversarial method for robust training of agents but only considered attacks on the dynamic of the environment, not on the visual perception of the agent. and developed adversarial environment generation to study agent's generalization and worst-case scenarios. Those are different from this present work where we enlighten how an adversary might take control of a neural policy. We wish to build an attack targeted towards the policy π target. At a time step t, the attack is said to be successful if π under attack indeed chooses the targeted action selected by π target. When π is not attacked, the attack success rate corresponds to the agreement rate between π and π target, measuring how often the policies agree along an unattacked trajectory of π. Note that we only deal with trained policies and no learning of neural policies is involved. In other words, π and π target are trained and frozen policies. What we really want to test is the ability of CopyCAT to lure π into having a specific behavior. For this reason, measuring the attack success rate is not enough. Having a high success rate does not necessarily mean the macroscopic behavior of the attacked agent matches the desired one as will be shown further in this section. Cumulative reward as a proxy for behavior We design the following setup. The agent has a policy π trained with DQN . The policy π target is trained with Rainbow . We select Atari games with a clear difference in terms of performance between the two algorithms (where Rainbow obtains higher average cumulative reward than DQN). This way, in addition to measuring the attack success rate, we can compare the cumulative reward obtained by π under attack ρ(π) to ρ(π target) as a proxy of how well π's behavior is matching the behavior induced by π target. In this setup, if the attacked policy indeed gets cumulative rewards as high as the ones obtained by π target, it will mean that we do not simply turned some actions into other actions we targeted, but that the whole behavior induced by π under attack matches the one induced by π target. This idea that, in reinforcement learning, cumulative reward is the right way to monitor an agent's behavior has been used and developed by the inverse reinforcement learning literature. Authors from argued that the value of a policy, i.e. its cumulative reward, is the most compact, robust and transferable description of its induced behavior. We argue that measuring cumulative reward is thus a reasonable proxy for monitoring the behavior of π under attack. At this point, we would like to carefully defuse a possible misunderstanding. Our goal is not to show that DQN's performance can be improved by being attacked. We simply want to show that its behavior can be fully manipulated by an opponent and we use the obtained cumulative reward as a proxy for the behavior under attack. Baseline We set the objective of building a real-time targeted attack. We thus need to compare our algorithm to baselines applicable ot this scenario. The fastest state-of-the-art method can be seen as a variation of. It applies targeted FGSM at each time step t to compute a new attack. It first infers the action a target and then back-propagates through the attacked network to compute their attack. CopyCAT only infers a target and then applies the corresponding pre-computed mask. Both methods can thus be considered usable in real-time yet CopyCAT is still faster at inference. We set the objective of attacking only observations rather than complete states so we do not need a write-access to the agent's inner workings. DQN stacks four consecutive frames to build its inner state. We thus compare CopyCAT to a version of the method from where the gradient inducing the FGSM attack is only computed w.r.t the last observation, so it produces an attack comparable to CopyCAT, i.e., on a single observation. The gradient from Eq. 1: ∇ st l(θ, s t, a target) becomes ∇ ot l(θ, f (o t, o 1:t−1), a target ). To keep the comparison fair, a target is always computed with the exact same policy π target as in CopyCAT. FGSM-L ∞ has the same parameter as CopyCAT, bounding the L ∞ norm of the attack. CopyCAT has an additional regularization parameter α allowing the attack to have, for a same, a lower energy and thus be less detectable. We will compare CopyCAT to the attack from showing how behaviors of π under attacks match π target when these attacks are of equal energy. Full optimization-based attacks would not be inferred fast enough to be used in a sequential decision making problem at each time step. Experimental setup We always turn the sticky actions on, which make the problem stochastic . An attacked observation is always clipped to the valid image range, 0 to 255. For Atari games, DQN uses as its inner state a stack of four observations: For learning the masks of ∆, we gather trajectories generated by π in order to fill D with 10k observations. We use a batch size of 8 and the Adam optimizer with a learning rate of 0.05. Each point of each plot is the average over 5 policy π seeds and 80 runs for each seed. Only one seed is used for π target to keep comparison in terms of cumulative reward fair. CopyCAT has an extra parameter α, we test its influence on the L 2 norm of the produced attack. For a given, FGSM-L ∞ computes an attack η of maximal energy. As given by Eq. 1, its L 2 norm is η 2 = √ N 2 with N the input dimension. For a given, CopyCAT produces |A| masks. We show in Fig. 1 the largest L 2 norm of the |A| masks for a varying α (plain curves) and compare it to the norm of the FGSM-L ∞ attack (dashed lines). We want to stress that the attacks are agnostic to the training algorithm so the are easily transferred to other agents using neural policies trained with another algorithm. As can be seen on Fig. 1, for a given and for the range of tested α, the attack produced by CopyCAT has lower energy than FGSM-L ∞. This is especially significant for higher values of, e.g higher than 0.05. Influence of parameters over the ing behavior We wish to show how the agent behaves under attack. As explained before, this analysis is twofold. First, we study in terms of attack success rate -rate of action chosen by π matching a target when shown attacked observations-as done in supervised learning. Second, we study the behavior matching through the cumulative rewards under attack ρ(π). What we wish to verify in the following experiment is CopyCAT's ability to lure an agent into following a specific behavior. If the attack success rate is high (close to 1), we know that, on a supervised-learning perspective, our attack is successful: it lures the agent into taking specific actions. If, in addition, the average cumulative reward obtained by the agent under attack reaches ρ(π target) it means that the attack is really successful in terms of behavior. We recall that we attack a policy with a target policy reaching higher average cumulative reward. We show on Fig. 2 and 3 (two different games) the attack success rate (left) and the cumulative reward (right) for CopyCAT (plain curves) for different values of the parameters α and, as well as for unattacked π (green dashed line) and π target (black dashed lines). We observe a gap between having a high success rate and forcing the behavior of π to match the one of π target. There seems to exist a threshold corresponding to the minimal success rate required for the behaviors to match. For example, as seen on the left, CopyCAT with = 5 and α < 10 −5 (green curve) is enough to get a 85% success rate on the attack. However, as seen on the right, it is not enough to get the behavior of π under attack to match the one of the target policy as the reward obtained under attack never reaches ρ(π target). Overall, we observe on Fig. 2-right and Fig. 3 -right that with high enough ≥ 0.04 and α < 10 −6, CopyCAT is able to consistently lure the agent into following the behaviour induced by π target. Comparison to We compare CopyCAT to the targeted version of FGSM on a setup where the gradient is computed only on the last observation. As in the last paragraph, we study both the attack success rate and the average cumulative reward under attack. We ask the question: is CopyCAT able to lure the agent into following the targeted behavior? Is it better at this task than FGSM in the real-time and read-only setting? We show on Fig. 4 and 5 (two different games) the success rate of CopyCAT and FGSM (y-axis, left) and the average cumulative reward under attack (y-axis, right). These values are plotted (i) against the L 2 norm of the attack for FGSM and (ii) against the largest L 2 norm of the masks: max i δ i 2 for CopyCAT. We only plot the standard deviation on the attack success rate because it corresponds to the intrinsic noise of CopyCAT. We do not plot it for cumulative reward for the reason that one seed of π target has a great variance (with the sticky actions) and matching π target, even perfectly, implies matching the variance of its cumulative rewards. The same phenomenon can be observed on Fig. 2 and 3: CopyCAT is not itself unstable (left figures, when α decreases or increases, the rate of successful attacks consistently increases). Yet the cumulative reward is noisier, as the behavior of π is now matching with a high-variance policy. As observed on Fig. 4-left and Fig. 5 -left, FGSM is able to turn a potentially significant part of the taken actions into the targeted actions (maximal success rate around 75% on Space Invaders). However, it is never able to make π's behavior match with π target's behavior as seen on Fig. 4-right and Fig. 5 -right. The average cumulative reward obtained by π under FGSM attack never reaches the one of π target. On the contrary, CopyCAT is able to successfully lure π into following the desired macroscopic behavior. First, it turns more than 99% of the taken actions into the targeted actions. Second, it makes ρ(π) under attack reach ρ(π target). Moreover, it does so using only a finite set of masks while the baselines compute a new attack at each time step. An example of CopyCAT is shown on Fig. 6. The patch δ i aiming at action "no-op" (i.e. do nothing) is applied to an agent playing Space Invaders. The patch itself can be seen on the right (gray represents a zero pixel, black negative and white positive). On the left, the unattacked observation. In the middle, the attacked observation. Below the images, the action taken by the same policy π when shown the different situations in an online setting. In this work, we built and showed the effectiveness of CopyCAT, a simple algorithm designed to attack neural policies in order to manipulate them. We showed its ability to lure a policy into having a desired behavior with a finite set of additive masks, usable in a real-time setting while being applied only on observations of the environment. We demonstrated the effectiveness of these universal masks in Atari games. As this work shows that one can easily manipulate a policy's behavior, a natural direction of work is to develop robust algorithms, either able to keep their normal behaviors when attacked or to detect attacks to treat them appropriately. Notice however that in a sequential-decisionmaking setting, detecting an attack is not enough as the agent cannot necessarily stop the process when detecting an attack and may have to keep outputting actions for incoming observations. It is thus an exciting direction of work to develop algorithm that are able to maintain their behavior under such manipulating attacks. Another interesting direction of work in order to build real-life attacks is to test targeted attacks on neural policies in the black-box scenario, with no access to network's weights and architecture. However, targeted adversarial examples are harder to compute than untargeted ones and we may experience more difficulties in reinforcement learning than supervised learning. Indeed, learned representations are known to be less interpretable and the variability between different random seeds to be higher than in supervised learning. Different policies trained with the same algorithm may thus lead to S → A mappings with very different decision boundaries. Transferring targeted examples may not be easy and would probably require to train imitation models to obtain mappings similar to π in order to compute transferable adversarial examples. In order to keep the core paper not too long, we only showed a subset of the in Sec. 5, they are all provided here. The explanations and interpretations can be found in Sec. 5. Left: HERO. Center: Space Invaders. Right: Air Raid. In this appendix, we provide additional experiments to study further various aspects of the proposed approach. In Sec. 5, the attacked agent was a trained DQN agent, while the target policy was a trained Rainbow agent. If these agents have clearly different behaviors, one could argue that they were initially trained to solve the same task (Rainbow achieving better ). To further assess CopyCAT's ability to lure a policy into following another policy, we therefore attack an untrained DQN, with random weights, to follow the policy π target (still obtained from a trained Rainbow agent). Left: HERO. Center: Space Invaders. Right: Air Raid. We see that in this case, FGSM is able to lure π into following π target at least as well as CopyCAT. This shows that it is easier to fool an untrained network than a trained one. As expected, trained networks are more robust to adversarial examples. CopyCAT is also able to lure the agent into following π target. B.2 observed the transferability of adversarial examples between different models. Thanks to this transferability, one is able to attack a model without having access to its weights. By learning attacks on proxy models, one can build black-box adversarial examples. enlightened the difficulty for the state-of-the-art methods to build targeted adversarial examples in this black-box setting. Starting from the intuition that universal attacks may transfer better between models, we enhanced CopyCAT for it to work in the black-box setting. We consider a setting where the adversary (i) is given a set of proxy models {π 1, ..., π n} trained with the same algorithm as π, (ii) can also query the attacked model π, but (iii) has no access to its weights. In the black-box setting, CopyCAT is divided into two steps: training multiple additional masks and selecting the highest performing ones. Training The ensemble-based method from computes its additional mask by attacking the classifier given by the mean predictions of the proxy models. We instead consider that our attack should be efficient against any convex combination of the proxy models' predictions. For each action a i, we compute the mask δ i by maximizing over 100 epochs on the dataset D: with ∆ the uniform distribution over the n-simplex. For each action, 100 masks are computed this way. These masks are just computed with different random seeds. Selection We then compute a competition accuracy for each of these random seeds. This accuracy is computed by querying π on states built as follows. We take four consecutive observations in D, apply 3 masks randomly selected among the previously computed masks on the first 3 observations; the mask δ i that is actually being tested is applied on the last observation. The attack is considered successful if π outputs the action a i corresponding to δ i. For each action, the mask with the highest competition accuracy among the 100 computed masks is selected. The selected masks are then used online as in the white-box setting. Results We provide preliminary for the considered black-box setting. Four proxy models of DQN are used to attack π. Again, it is attacked to make it follow the policy π target given by Rainbow. The can be found in Fig. 8. Each dot is an attack tested over 80 new episodes. Y-axis is the mean success rate (middle) or the cumulative reward (right). X-axis is the maximal norm of the attack. The figure on the left gives the value of α (on the y-axis) corresponding to each color. We can observe that the proposed black-box attack is effective, even if less efficient than its white-box counterpart. The proposed black-box CopyCAT could certainly be improved, and we let this for future work. Reinforcement learning led to great improvements for games or robots manipulation but is not able yet to tackle realistic-image environments. While this paper is focused on weaknesses of reinforcement learning agents, the relevance of the proposed method would be diminished if one could not compute universal adversarial examples on realistic datasets. We thus present this proof-of-concept, showing the existence of universal adversarial examples on ImageNet . Note that already showed the existence of universal attacks but considers a patch covering a part of the image rather than an additional mask. We computed a universal attack on VGG16 , targeted towards the label "tiger shark", the same way CopyCAT does. It is trained on a small training set (10 batches of size 8), and tested on a random subset of ImageNet validation dataset. The network is taken from Keras pretrained models and attacked in a white-box setting. The same procedure as CopyCAT is used. 1000 images are randomly selected in the validation set. Only 80 are used for training and the rest is used for testing. The attack is trained with the same loss, same learning rate and same batch size as CopyCAT, for 200 epochs. The (rescaled) computed attack is shown in Fig. 9. Examples of attacked images from the test set are visible on Fig. 10. After 200 epochs, the train accuracy is 90% and the test accuracy 88.44%. This proof-of-concept experiment validates the existence of universal adversarial examples on realistic images and shows that CopyCAT's scope is not reduced to Atari-like environments. More generally, the existence of adversarial examples have been shown to be a property of high-dimensional manifolds . Going towards more realistic images, hence higher dimensional images, should on the opposite, allow CopyCAT to more easily find universal adversarial examples."

Simple generative approach to solve the word analogy problem which yields insights into word relationships, and the problems with estimating them

Represent this text to find a much much longer article about it with say hundreds-thousands of words

The choice of the hub (target) language affects the quality of cross-lingual embeddings, which shouldn't be evaluated only  on English-centric dictionaries.