Question: Represent the scientific article to find a short TLDR

Question: The ability of algorithms to evolve or learn (compositional) communication protocols has traditionally been studied in the language evolution literature through the use of emergent communication tasks. Here we scale up this research by using contemporary deep learning methods and by training reinforcement-learning neural network agents on referential communication games. We extend previous work, in which agents were trained in symbolic environments, by developing agents which are able to learn from raw pixel data, a more challenging and realistic input representation. We find that the degree of structure found in the input data affects the nature of the emerged protocols, and thereby corroborate the hypothesis that structured compositional language is most likely to emerge when agents perceive the world as being structured. The study of emergent communication is important for two related problems in language development, both human and artificial: language evolution, the development of communication protocols from scratch BID27; and language acquisition, the ability of an embodied agent to learn an existing language. In this paper we focus on the problem of how environmental or pre-linguistic conditions affect the nature of the communication protocol that an agent learns. The increasing realism and complexity of environments being used for grounded language learning BID8 BID18 present an opportunity to analyse these effects in detail. In line with previous work on emergent communication, we are strongly motivated by the view that language derives meaning from its use BID39 BID37. This perspective especially motivates the study of language emergence in cases where co-operative agents try to achieve shared goals in game scenarios BID34 BID6 BID26, and is related to the study of multi-agent and self-play methods that have found great success in other areas of machine learning BID1 BID30. Here we focus on simple referential games, in which one agent must communicate to another a target object in the agent's environment. One of the most important properties of natural language is compositionality. Smaller building blocks (e.g. words, morphemes) are used to generate unbounded numbers of more complex forms (e.g. sentences, multi-word expressions), with the meaning of the larger form being determined by the meanings of its parts and how they are put together BID14. Compositionality is an advantage in any communication protocol as it allows in principle infinite expression through a finite dictionary and a finite set of combination rules. In emergent communication research, previous work has shown that agents can produce (somewhat) compositional protocols when engaging in language games BID34. However, the computational agents were typically situated in artificial worlds containing just a handful of objects, represented as disentangled, structured, and sometimes even atomic symbols, e.g. attribute-based or one-hot vectors BID2 BID5 BID13 BID0 BID26. However, humans receive raw sensorimotor hank you! h a n k y o u! rather than symbolic input, and little work to date has tested whether these findings carry over when agents are situated in less idealized worlds that bear more similarity to the kind of entangled and noisy environments to which humans are typically exposed. In this work, in the context of referential communication games (see FIG0, we contrast the of two studies that lie at the extremes of how much structure is provided by the environment. The first study (Section 3) focuses on symbolic representations, where objects are represented as bags-of-attributes; this representation is inherently disentangled since dimensions encode individual properties. The second study (Section 4) considers raw perceptual input, hence data that more closely resembles what humans are exposed to. Clearly, the latter is a more challenging and realistic scenario as the computational agents are operating on entangled inputs with no pre-coded semantics. Crucially, both studies use the same referential game setup, the same learning procedure (policy learning methods) and the same neural network agent architectures. We show that reinforcement learning agents can successfully communicate, not only when presented with symbolic and highly structured input data, but (and more importantly) even when presented with raw pixel input. This opens up the possibility of more realistic simulations of language emergence. We successfully use the learning signal from the referential game to train agents end-to-end, including cases where the agents need to perform visual processing of images with a convolutional neural network. However, we find that the agents struggle to produce structured messages when presented with entangled input data BID3 due to the difficulty of uncovering the true factors of variation, corroborating the hypothesis of BID32 that structured (compositional) language is most likely to emerge when agents perceive the world as structured. The referential game is implemented as an instance of multi-agent co-operative reinforcement learning, in which two agents take discrete actions in their environment in order to maximize a shared reward. The referential game is a variant of the Lewis signaling game BID23, which has been extensively used in linguistic and cognitive studies in the context of language evolution (e.g., BID7 ; BID35 BID33 BID22 .Figure 1 provides a schematic description of our setup. First, a speaker is presented with a target object (highlighted as CAR in the symbolic example on the left, and highlighted as the far right image in the pixel example on the right). Then, by making use of an alphabet consisting of primitive discrete symbols ("22", "10", "0","2"), the speaker constructs a message describing that object ("22 2 0"). We will refer to the set of all distinct messages generated by the speaker as their lexicon or protocol. Finally, the listener is presented with the target and a set of distractor objects, and-by making use of the speaker's message-has to identify the target object from the set of candidate objects. Communicative success is defined as the correct identification of the target by the listening agent. Formally, the attribute-based object vectors (disentangled) or the pixel-based images (entangled) are the set of pre-linguistic items W = {o 1, . . ., o N}. From this set we draw a target t ∈ W and subsequently DISPLAYFORM0 The speaker has only access to the target t, while the listener receives candidate set C = t ∪ D, not knowing which of the elements in C is target t. The speaker encodes t into a dense representation u using an encoder f S (t, θ S f). The function of this encoder depends on the type of pre-linguistic data used and is discussed separately for each study. Given an alphabet A of discrete unit symbols (akin to words) and u, the speaker next generates a discrete, variable-length, bounded message m by sampling symbols from a recurrent policy π DISPLAYFORM0 ). The sequence generation is terminated either by the production of a stop symbol or when the maximum length L has been reached. We implement the decoder as a single-layer LSTM BID19. Note that the symbols in the agents' alphabet A have no a priori meaning; rather, these symbols get grounded during the game. The listening agent uses a similar encoder to the speaker but has independent network weights (θ L f). Applying this encoder to all candidate objects in a set DISPLAYFORM1 For encoding the message m, we use a single-layer LSTM, denoted h L, which produces an encoding z: DISPLAYFORM2 ). Given encoded message z and candidates U, the listener predicts a target object t ∈ C following a policy π L implemented using a non-parametric pointing module; this module samples the predicted object from a Gibbs distribution computed via the dot product between vector z and all encoded candidates u ∈ U. See Appendix B for information regarding the agents' architecture. At inference time, we replace the stochastic sampling of the speaker's message and the listener's stochastic pointing module with deterministic processes. For the pointing module, the object with the highest probability is chosen. For the speaker's message, this is generated in a greedy fashion by selecting the highest-probability symbol at each step. All weights of the speaker and listener agents, θ = {θ DISPLAYFORM0 are jointly optimized while playing the game. We emphasize that no weights are shared between the speaker and the listener, and the only supervision used is communicative success, i.e. whether the listener identified the correct target. The objective function that the two agents maximize for one training instance is: DISPLAYFORM1 where R is the reward function returning 1 if t = t (if the listener pointed to the correct target) and 0 otherwise. To maintain exploration in the speaker's policy π S of generating a message, and the listener's policy π L of pointing to the target, we add to the loss an entropy regularization term BID25. The parameters are estimated using the REINFORCE update rule BID38 Table 1: Commumicative success (training accuracy in percentage) with varying maximum message length. alphabet size denotes the effective size of the symbol set used from a maximum of 100. lexicon size is the effective number of unique messages used. topographic ρ reports the structural similarity in terms of Spearman ρ correlation between the message and the object vector space. All Spearman ρ correlations throughout the paper are significant with p < 0.01. We first present experiments where agents are learning to communicate when presented with structured and disentangled input. We use the Visual Attributes for Concepts Dataset (VisA) of BID29, which contains human-generated per-concept attribute annotations for 500 concrete concepts (e.g., cat, sofa, car) spanning across different categories (e.g., mammals, furniture, vehicles), annotated with 636 general attributes (e.g., has tail, is black, has wheels). We disregarded homonym concepts (e.g., bat), thus reducing our working set of concepts to 463 and the number of attributes to 573 (after eliminating any attribute that did not occur with the working concepts). On average, each concept has 11 attributes. All pre-linguistic objects are represented in terms of binary vectors o ∈ {0, 1} 573. Note that these representations do carry some inherent structure; the dimensions in the object vectors are disentangled and so each object can be seen as a conjunction of properties. Speaker and listener convert the pre-linguistic representations to dense representations u by using a single-layer MLP with a sigmoid activation function. In all experiments, we set the number of candidate objects K to five, meaning there were four wrong choices per correct one (ing in a 20% random baseline). Inspired by BID21, who show that non-compositional language emerges in the case of overcomplete alphabets, we set the size of alphabet A to 100 symbols, which is smaller than the size of the set of objects. We first report model performance on the training data, comparing different settings for the maximal allowed message length (2, 5 or 10 symbols). Results are presented in Table 1 (ignore the last row topographic ρ which will be explained in later sections).In the case of the shortest message settings (maximum length 2), our trained agents on average only develop a protocol of 31 unique messages used to describe 363 training concepts (leaving aside 100 for testing). This indicates high levels of ambiguity, with each message being used to denote 11 concepts on average. Interestingly, recent findings suggest that ambiguity is a design feature of language that prevents the inefficient use of redundant codes, since some of the message content can be extracted from context: "the most efficient communication system will not convey information already provided by the context" BID28. In our case, we do no explicitly encode any bias towards ambiguity. We hypothesize that ambiguity arises due to the difficult exploration problem that agents are faced with, in combination with the fact that ambiguous protocols present a good local optimum that is over-represented in the hypothesis search space. As a , in the absence of environmental pressures (e.g., a high number of carefully constructed distractors) a suboptimal policy can still achieve a reasonably high accuracy (92%), making it even harder during training to escape from such a solution. In classic signaling games, this polysemy phenomenon manifests itself as different states receiving the same signal and is termed partial pooling equilibrium BID31. Perhaps rather counterintuitively, Skyrms (p.131) suggests that a way to obtain communication protocols that are robust to this type of local communication minima is to allow the invention of new signals, essentially increasing the search space of signals. Motivated by this suggestion, we play variants of the game in which we allow the agents to produce messages of greater maximum length (5 and 10), which leads to improved communicative success (98.2% and 98.5% respectively). We observe that the number of messages in the protocol increases from 31 to 293 and 355, respectively, reducing the average number of concepts a message can denote from 11 concepts to (approximately) 1 concept. In the real world, when speakers refer to cats, listeners would likely be in a situation where they had to discriminate a cat in the context of a couch or a dog, rather than in the context of a mirror or a cow.2 Simply put, objects in the world do not appear in random contexts, but rather there is regularity in the distribution of situational and visual co-occurrences. This property of the world is typically not captured in referential games studied in the language emergence literature, with distractors usually drawn from a uniform distribution. We address this issue and design an additional experiment with distractors sampled from a targetspecific context distribution reflecting normalized object co-occurrence statistics. Co-occurrence data is extracted from the MSCOCO caption dataset BID24. This leads to more plausible distractor sets with, for instance, the target goat more likely being mixed with sheep and cow as distractors rather than bike or eggplant. We find that the distractor selection process (uniform vs context-dependent) affects the language learning dynamics; see FIG1 for training curves for different experimental configurations. While the non-uniform distractor sampling of the context-dependent setting can be exploited to learn a degenerate strategy -giving up to 40% communicative success shortly after the start of trainingsubsequently learning under this scenario takes longer. This effect is likely a combination of the local minimum achieved by the degenerate strategy of picking a target at random from only the topically relevant set of distractors, which initially makes the problem easier; however, the fact that the co-occurrence statistics tend to align with the feature vectors, means that similar objects are more likely to appear as distractors and hence the overall game becomes more difficult. We now consider the question of how objects denoted by the same (ambiguous) message are related. When the context is drawn uniformly, object similarity is a predictor of object confusability, as similar objects tend to be mapped onto the same message (0.26 and 0.43 median pairwise cosine similarities of objects that received the same message as computed on the VisA space, for maximum message length 2 and 5, respectively). In the non-uniform case, we observe object confusability to be less influenced by object similarity (0.15 and 0.17 median pairwise cosine similarities of objects that received the same message, for maximum message length 2 and 5, respectively), but rather driven by the visual context co-occurrences. Simply put, in the non-uniform case confusability is less influenced by similarity since the agents must learn to distinguish between objects that naturally co-occur (e.g. sheep and goat). Thus, the choice of distractors, an experimental design decision that in existing language emergence literature has been neglected, has an effect on the organization (and Table 2 : Communicative success (acc in percentage) of agents evaluated on training (first row) and novel (last three rows) data. lexicon size column reports the percentage of novel messages (i.e., messages that were not used during the training).potentially the naturalness) of the emerged language, for example as reflected in the semantics of ambiguous or homonym words in the language. Quantifying the degree of compositionality and structure found in the emerged language is a challenging task; to the best of our knowledge, there is no formal mathematical definition of compositionality that would allow for a definitive quantitative measure. Thus, research on this topic usually relies on defining necessary requirements that any language claiming to be compositional should adhere to, such as the ability to generalize to novel situations BID2 BID13 BID21. We adopt a similar strategy by measuring the extent to which an emerged language is able to generalize to novel objects (Section 3.3.1). Moreover, we also report quantitative (Section 3.3.2) using a measure of message structure proposed in the language evolution literature BID6 BID10. We perform experiments where trained agents from Section 3.1 are exposed to different types of unseen objects, each of them differing to the degree to which the unseen objects resemble the objects found in the training data. In the test scenario, objects come from the same data distribution as the training data, but were not presented to the agents during training (e.g., a mouse); in the unigram chimeras scenario, the novel objects are constructed by sampling properties from a property-based distribution inferred from the training data, thus breaking any feature correlation (e.g., a mouselike animal with wheels); in the uniform chimeras scenario, the novel objects are constructed by uniformly sampling properties (e.g., a square red furry metallic object). Table 2 reports the communicative success. While there is a drop in performance for unseen objects, agents are performing above random chance (20%). The emerged language is indeed able to generalize to unseen objects; however, the degree of generalization is a function of the similarity between the training and unseen objects, thus ing in the uniform chimeras setting obtaining the lowest performance. Moreover, we observe examples of productivity, a key feature of compositionality. At test time, speakers are able to concoct novel messages on-the-fly (i.e., messages that are not part of their lexicon induced during training) to describe unseen objects. See the last three rows of Table 2, and the lexicon size column, for the percentage of novel messages. Even though listeners were not trained to associate novel messages with novel objects, they are still able comprehend such messages and correctly identify the target object. In the test data and length 10 cases, novel messages account for almost all of the generated messages, but with performance at 81.6%, providing evidence of the structure found in the messages. Given a set of objects, their meanings and the associated signals, BID6 define topographic similarity to be the correlation of the distances between all the possible pairs of meanings and the corresponding pairs of signals. compositional languages being higher than that of holistic. The intuition behind this measure is that semantically similar objects should have similar messages. To compute this measure, we first compute two lists of numbers: (i) the Levenshtein distances between all pairs of objects' messages; and (ii) the cosine similarity between all pairs of objects' VisA vectors. Given these two lists, the topographic similarity is defined as their negative Spearman ρ correlation (since we are correlating distances with similarities, negative values of correlation indicate topographic similarity of the two spaces). Intuitively, if similar objects share much of the message structure (e.g., common prefixes or suffixes), and dissimilar objects have little common structure in their respective messages, then the topographic similarity should be high, the highest possible value being 1.Results presented back in Table 1, in the topographic ρ column, show that topographic similarity is positive in all experimental setups, indicating that similar objects receive similar messages (p < 0.01, permutation test). A qualitative analysis of the messages generated in the length 10 and training data cases showed that, for example, 32% of the mammal objects had as a message prefix the bigram'95#10'; 36% of vehicle objects had'68#95'; and 11% of tool objects had'0#61', suggesting that these prefix bigrams encode category-specific information. Next, for each object pair, we calculate their Levenshtein message distance and respective cosine similarity, and plot in Figure 3 (right), for each distance, the average cosine similarities of the pairs with that distance (this is done for the length 10 and training data experiment). We observe that there is a clear relation between message similarity and meaning similarity (as measured by overlap in the VisA properties). In Figure 3, we also plot a similar correlation curve for an emerged language obtained by producing messages with randomly initialized and untrained speaker/listener architectures. This emerged language is at random in terms of communicative success; however, the generated messages do show signs of structure, since similar objects obtain somewhat similar messages. This seems to suggest that structured and disentangled pre-linguistic representations are, perhaps, a sufficient condition for the emergence of structured language, especially in neural network-based agents which, due to the nature of representation and information flow, favor similar inputs to trigger similar outputs. In this section, we present experiments in which agents receive as input entangled data in the form of raw pixel input, and have to learn to perform visual conceptual processing guided by the communication-based reward. We use a synthetic dataset of scenes consisting of geometric objects generated using the MuJoCo physics engine BID36. We generate RGB images of resolution 124 × 124 depicting single object scenes. For each object, we pick one of eight colors (blue, red, white, black, yellow, green, cyan, magenta) and five shapes (box, sphere, cylinder, capsule, ellipsoid) ing in 40 combinations, for each of which we generate 100 variations, varying the floor color and the object location in the image. Moreover, we introduce different variants of the game: game A with 19 distractors; game B with 1 distractor; game C with 1 distractor, and with speaker and listener having different viewpoints of the target object (the target object on the listener's side is in a different location); game D with 1 distractor, with speaker and listener having different viewpoints, and with balanced numbers of shapes and color (obtained by downsampling from 8 colors to 5 and removing any image containing objects of the 3 disregarded objects). For each game, we create train and test splits with proportions 75/25 (i.e., 3000/1000 for games A and B, and 1850/650 for games C and D).Pre-linguistic objects are presented in the form of pixel input, o ∈ 3×124×124. Speaker and listener convert the images o to dense representations u, each of them using an 8-layer convolutional neural network (ConvNet). Crucially, we do not pre-train the ConvNets on an object classification task; the only learning signal is the communication-based reward. Despite this fact, we observe that the lower layers of the ConvNets are encoding similar information to a ConvNet pre-trained on ImageNet BID11.3 Conceptually, we can think of the whole speaker/listener architecture as an encoder-decoder with a discrete bottleneck (the message). Given our initial positive findings, this reward-based learning signal induced from the communication game setup could be used for classagnostic large-scale ConvNet training. Moreover, we find that, even though no weights were shared, the agents' conceptual spaces get aligned at different levels, reminiscent of theories of interactive conceptual alignment during dialogue BID15 ) (see Appendix A for the related experiment). Unlike the experiments of Section 3, where agents start from disentangled representations, starting from raw perceptual input presents a greater challenge: the agents have to establish naming conventions about scenes, while at the same time learning to process the input with their own visual conceptual system. Since we do not pre-train their ConvNets on an object recognition task, the dense representations u used to derive the message contain no bias towards any image-or scene-specific information (e.g, object color, shape or location). The extraction of visual properties is thus driven entirely by the communication game. This contrasts with the cases of BID16 and BID22 who use pre-trained visual vectors, and qualitatively observe that the induced communication protocols encode information about objects. Table 3 presents the in terms of communicative train and test success (see Appendix C for additional experiments when having access to gold object attribute classifiers). Moreover, we also report the topographic similarity (column topographic ρ) between the symbolic attribute-based representations of scenes (floor color, object color, shape and location) and the generated messages. Overall, despite the challenges posed in this setup due to the raw nature of the data, performance across all games is well above chance, indicating that reinforcement learning agents trained end-toend are able to establish a communication protocol in this grounded environment. In game A, the agents reach 93.7% accuracy, with their lexicon consisting of 1068 messages, describing 3000 training objects. Most importantly, as captured by the positive topographic similarity, agents produce messages that respect (even to a limited degree) the compositional nature of scenes (i.e., objects as bags-of-attributes), indicating that similar scenes receive similar messages. Indeed, by examining their protocol (see Table 4), we find that messages encode in a structurally consistent way information about absolute location of objects, with the message prefix and suffix denoting the horizontal and vertical co-ordinate, respectively. Interestingly, this communication strategy is also typically followed by human players of referential games BID20. 3 Specifically, for all 4000 images we compute two sets of activations, one derived from the speaker's ConvNet and one from a pre-trained ResNet model BID17. We then compute all pairwise cosines in the speaker's ConvNet and ResNet space and correlate these values. We find Spearman ρ to be in the range 0.6-0.7 between the first 3 layers of the speaker's ConvNet and the ResNet. Table 3: Communicative success of agents playing different games. Columns random, train and test report percentage accuracies. Column topographic ρ reports the topographic similarity between the symbolic representation of scenes and the generated messages (p < 0.01, permutation test).However, we find the emerged protocols to be very unstable and too grounded in the specific game situation. Small modifications of the game setup, while having close to no negative impact on the communicative performance, can radically alter the form, semantics and interpretability of the communication protocol. In game B, performance remains at the same level (93.2%) as game A. However, we observe that the protocol consists of 13 unique messages which do not reflect the objects' attributes (as indicated by the close to zero topographic similarity), thus making the messages harder to interpret (see FIG3 for randomly sampled examples). When we change the viewpoint of the agents in game C, biasing them against communicating about absolute object location, the players derive a compact communication protocol consisting of 8 unique messages that describe primarily color. Finally, when color and shape are balanced, as in game D, we still observe a bias towards describing the color of objects, with the five induced messages providing a perfect clustering of the objects according to their colors. Table 4: Accuracy of probe linear classifiers of speaker's induced visual representations (all accuracies are in percentage format). In order to investigate what information gets captured by the speaker's ConvNet, we probe the inferred visual representations u used to derive the message. Specifically, we design 4 probe classifiers for the color and shape of the object; object position which is derived by discretizing each co-ordinate into 3 bins; and floor color which is obtained by clustering the RGB color representation of the floor. For each probe, we performed 5-fold cross validation with a linear classifier, and report accuracy in Table 4. Overall, different games in visual representations with different predictive power; object position is almost always encoded in the speaker's visual representation, even in situations where location of the object is not a good strategy for communication. On the other hand, object shape seems to provide less salient information, despite the fact that it is relevant for communication, at least in the C&D games. As expected, the structure and semantics of the emergent protocols are a function of the information captured in the visual representations. The degree to which the agents are able to pull apart the objects' factors of variation impacts their ability to communicate about those factors, with the most extreme case being game D, where the message ignores the shape entirely. Thus, disentanglement seems to be a necessary condition for communication, at least in the case of pixel input. We presented a series of studies investigating the properties of protocols emerging when reinforcement learning agents are trained end-to-end on referential communication games. We found that when agents are presented with disentangled input data in the form of attribute vectors, this inherent compositional structure is successfully retained in the output. Moreover, we showed that communication can also be achieved in cases where agents are presented with raw pixel data, a type of input that aligns better with the raw sensorimotor data that humans are exposed to. At the same time, we found that their ability to form compositional protocols in these cases is hampered by their ability to pull apart the objects' factors of variations. Altogether, we were able to successfully scale up traditional research from the language evolution literature on emergent communication tasks to the contemporary deep learning framework, thus opening avenues to more realistic, and large scale, computational simulations of language emergence with complex image stimuli. During conversation, communication allows interlocutors to achieve interactive conceptual alignment BID15. We are able to communicate because we have established a common ground and our representations at different levels become aligned (e.g., participants mutually understand that "he" in the conversation refers to Bob). We investigated whether the agents' conceptual systems achieve a similar structural alignment. We measure the alignment in terms of Spearman ρ correlation of the intra-agent pairwise object cosine similarities as calculated via representing objects as activations from ConvNet layers. Interestingly, we observe a gradual increase in the structural similarity as we represent the objects with layer activations closer to the pixel space. Conceptual spaces are more aligned the closer they are to the raw pixel input (ρ = 0.97-0.91, depending on the game) and become more dissimilar as the representations become more abstract. We can draw the analogy to language processing, as first ConvNet layers perform some low-level processing analogous to phoneme recognition or word segmentation (and are thus more objective) while higher layers perform more abstract processing, vaguely analogous to semantics and pragmatics (thus, represent more subjective knowledge). In cases of successful communication, speakers' and listeners' conceptual spaces closer to the communication point are structurally very similar (ρ = 0.85-0.62, depending on the game), however this similarity drops dramatically in cases of failure of communication (ρ = 0.15). All LSTM hidden states of the "speaking" and "listening" module as well and the "seeing" prelinguistic feed-forward encoders (see Section 3), have dimension 50. The "seeing" pre-linguistic ConvNet encoders (see Section 4) has 8 layers, 32 filters with the kernel size 3 for every layer and with strides for each layer. We use ReLU as activation function as well as batch normalization for every layer. For learning, we used the Rmsprop optimizer, with learning rate 0.0001. We use a separate value of entropy regularization for each policy. For π S we use 0.01 and for π L we use 0.001. We use a mini-batch of 32. We assume a model which has access to perfect attribute classifiers for color, shape and object position, for the latter using a classifier operating on the discretized annotations we obtained in Section 4.2 after quantazing the real-valued object location. For computing the performance of this model using gold attribute classifiers, we first remove from the distractors any candidate not matching the target's attributes and them pick at random. We repeat this experiment for single attribute classifiers and their pairwise combinations. TAB4: Communicative success of trained models from Section 4.1 (train and test) as well as models with access to gold classifiers. All accuracies are in percentage format.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

A controlled study of the role of environments with respect to properties in emergent communication protocols.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR.

Question: For understanding generic documents, information like font sizes, column layout, and generally the positioning of words may carry semantic information that is crucial for solving a downstream document intelligence task. Our novel BERTgrid, which is based on Chargrid by , represents a document as a grid of contextualized word piece embedding vectors, thereby making its spatial structure and semantics accessible to the processing neural network. The contextualized embedding vectors are retrieved from a BERT language model. We use BERTgrid in combination with a fully convolutional network on a semantic instance segmentation task for extracting fields from invoices. We demonstrate its performance on tabulated line item and document header field extraction. Documents often come in a variety of layouts and formats. For instance, a single document may contain isolated text boxes, tabular arrangements, multiple columns, and different font sizes. This layout can carry crucial semantic information. In classical natural language processing (NLP), however, the layout information is completely discarded as the document text is simply a sequence of words. Without access to the layout, a downstream task such as extraction of tabulated data can become much harder -and in some cases impossible to solve -since the necessary serialization may lead to severe information loss. Instead of working on the textual level, it is possible to directly apply methods from computer vision (CV) (e.g.) to work on the raw document pixel level which naturally retains the two-dimensional (2D) document structure. However, this is impractical, as a machine learning model would first need to learn textual information from the raw pixel data followed by the semantics. Recent approaches have designed a hybrid between NLP and CV methods for document intelligence: Chargrid , followed more recently by CUTIE , construct a 2D grid of characters or words from a document and feed it into a neural model, thereby preserving the spatial arrangement of the document. The symbols in the original document are embedded in some vector space, yielding a rank-3 tensor (width, height, embedding). Both papers report significant benefits of using such a grid approach over purely sequential 1D input representations, especially for semantically understanding tabulated or otherwise spatially arranged text like line items. With our contribution BERTgrid, we incorporate contextualized embedding into the grid document representation. More specifically, we use a BERT language model pre-trained on a large pool of unlabeled documents from the target domain to compute contextualized feature vectors for every word piece in a document. We demonstrate the effectiveness of BERTgrid on an invoice information extraction task from document tables and headers. We compare our to Chargrid and find significant improvements from 61.76% ± 0.72 to 65.48% ± 0.58 on an invoice dataset previously described in. Instead of constructing a grid on the character level and embedding each character with one-hot encoding as in , we construct a grid on the word-piece level and embed with dense contextualized vectors from a BERT language model. Formally, let a document be denoted by max | j ∈ {1, . . ., n}, consisting of n word pieces w (j), each of which is associated with a non-overlapping bounding... w (n) be the line-by-line serialized version of D. Using all word pieces j ∈ {1, . . ., n}, the BERTgrid representation of the document is defined as where d is the embedding dimensionality, e is the embedding function, and 0 d denotes an all-zero vector which we use for . We implement e using a pre-trained BERT language model. During evaluation of e, S is fed into the BERT model. The representation of the second-to-last hidden layer for the jth position is used to embed w (j) at position x, y in W. Fig. 2 (d) visualizes a BERTgrid tensor W. Our model pipeline is summarized in Fig. 1. A raw document image is first passed through an OCR engine to retrieve the words and their positions, i.e. D. We then serialize D ing in S which is subsequently passed through a BERT language model (BERT BASE configuration from) to get a contextualized vector for each word. Together with positional information from D, we construct W according to Eq. 1. For our downstream information extraction task, we use the same fully convolutional encoder-decoder neural network architecture and the same semantic segmentation and bounding box regression training tasks as , except W is the input to the neural network. Just like in , we obtain extracted document strings by comparing the predicted segmentation mask and bounding boxes with D. We interchangeably use BERTgrid for denoting just the document representation or the complete model consisting of input and network. As an extension to BERTgrid, we also construct a second model [C+BERTgrid] which combines the Chargrid and BERTgrid input representations. For that, we replicate the first convolutional block of the neural network to have a Chargrid and a BERTgrid branch. Both are subsequently merged by adding the two hidden representations. All models are trained for 800k iterations on a single Nvidia V100 GPU each. The BERT model with sequence length 512 is pre-trained for 2M steps and not fine-tuned on the downstream task. 3 Experiments As a concrete example for document intelligence, we extract key-value information from invoices without making assumptions on the invoice layout. We distinguish two kinds of fields: header fields and line item fields. The former includes invoice amount, invoice number, invoice date, and vendor name and address. The latter includes line item quantity, description, VAT amount/rate, and total price. It is usually contained in tabulated form and can occur in multiple instances per invoice. For each line item, all associated fields are grouped by a single bounding box per line item. Not all fields are always present on an invoice. We use the same dataset for training and testing as described in. It is comprised of 12k samples which we split 10k/1k/1k for training/validation/testing. The invoices are from a large variety of different vendors and the sets of vendors contained in training, validation, and testing samples are disjoint. Languages are mixed, with the majority being English. An example invoice along with its ground truth annotations is shown in Fig. 2. In addition, we use a second, much larger dataset comprised of 700k unlabeled invoices. This dataset is serialized ing in about 800 MB of plain text data. We use it for pre-training BERT from scratch as well as learning embedding vectors with word2vec. We use the evaluation metric from. This measure is similar to the edit distance, a measure for the dissimilarity of two strings. For a given field, we count the number of insertions, deletions, and modifications of the predicted instances to match the ground truth (pooled across the entire test set). The measure is computed as where N is the total number of instances of a given field occurring in the ground truth of the entire test set. This measure can be negative, meaning that it would be less work to perform the extraction manually. The best value it can reach is 1, corresponding to perfect extraction. Tab. 1 shows the in terms of the evaluation measure for different input representations. All are averaged over four randomly initialized training runs.; have shown that grid-based approaches like [Chargrid] or [Wordgrid] outperform conventional sequential models as well as purely image-based methods, so we use [Chargrid] as our baseline, with 61.76% ± 0.72. We assume the performance of BERTgrid stems from (i) embedding on the word-piece level and (ii) contextualization. Rather than learning to represent words first, the network directly gets access to semantically meaningful word(-piece)-level information. For instance, words such as avenue, street, and drive are very different when embedded on the character level, but will be mapped to approximately the same embedding vector. We observe that both [C+Wordgrid] and [C+BERTgrid] converge faster than [Chargrid] which supports this statement. During language model pre-training on the large, unlabeled dataset, knowledge about the language of invoices is distilled into the BERT model parameters. Compared to simpler, non-contextualized embedding methods such as word2vec, it has sufficient capacity to capture complex dependencies. This distilled knowledge is made accessible via the BERTgrid representation and eases the downstream task significantly. We acknowledge the BERT model has only access to S, not D. Future work could use 2D positional encodings to preserve the layout structure also during language model pre-training and inference.

Represent the short summary for finding the corresponding scientific article (long)

Grid-based document representation with contextualized embedding vectors for documents with 2D layouts

Represent the TLDR for finding the corresponding scientific article


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Videos are available at https://attackingrl.github.io. The discovery of adversarial examples for image classifiers prompted a new field of research into adversarial attacks and defenses . Recent work has shown that deep RL policies are also vulnerable to adversarial perturbations of image observations ). However, real-world RL agents inhabit natural environments populated by other agents, including humans, who can only modify observations through their actions. We explore whether it's possible to attack a victim policy by building an adversarial policy that takes actions in a shared environment, inducing natural observations which have adversarial effects on the victim. RL has been applied in settings as varied as autonomous driving , negotiation and automated trading . In domains such as these, an attacker cannot usually directly modify the victim policy's input. For example, in autonomous driving pedestrians and other drivers can take actions in the world that affect the camera image, but only in a physically realistic fashion. They cannot add noise to arbitrary pixels, or make a building disappear. Similarly, in financial trading an attacker can send orders to an exchange which will appear in the victim's market data feed, but the attacker cannot modify observations of a third party's orders. Figure 1: Illustrative snapshots of a victim (in blue) against normal and adversarial opponents (in red). The victim wins if it crosses the finish line; otherwise, the opponent wins. Despite never standing up, the adversarial opponent wins 86% of episodes, far above the normal opponent's 47% win rate. environments are substantially more vulnerable to adversarial policies than in lower-dimensional Ant environments. To gain insight into why adversarial policies succeed, we analyze the activations of the victim's policy network using a Gaussian Mixture Model and t-SNE . We find adversarial policies induce significantly different activations than normal opponents, and that the adversarial activations are typically more widely dispersed across time steps than normal activations. A natural defence is to fine-tune the victim against the adversary. We find this protects against that particular adversary, but that repeating the attack method finds a new adversary the fine-tuned victim is vulnerable to. However, the new adversary is qualitatively different, physically interfering with the victim. This suggests repeated fine-tuning might provide protection against a range of adversaries. Our paper makes three contributions. First, we propose a novel, physically realistic threat model for adversarial examples in RL. Second, we demonstrate the existence of adversarial policies in this threat model, in several simulated robotics games. Our adversarial policies reliably beat the victim, despite training with less than 3% as many timesteps and generating seemingly random behavior. Third, we conduct a detailed analysis of why the adversarial policies work. We show they create natural observations that are adversarial to the victim and push the activations of the victim's policy network off-distribution. Additionally, we find policies are easier to attack in high-dimensional environments. As deep RL is increasingly deployed in environments with potential adversaries, we believe it is important that practitioners are aware of this previously unrecognized threat model. Moreover, even in benign settings, we believe adversarial policies can be a useful tool for uncovering unexpected policy failure modes. Finally, we are excited by the potential of adversarial training using adversarial policies, which could improve robustness relative to conventional self-play by training against adversaries that exploit weaknesses undiscovered by the distribution of similar opponents present during self-play. Most study of adversarial examples has focused on small p norm perturbations to images, which discovered cause a variety of models to confidently mispredict the class, even though the changes are visually imperceptible to a human. Gilmer et al. (2018a) argued that attackers are not limited to small perturbations, and can instead construct new images or search for naturally misclassified images. argue that the near-ubiquitous p model is merely a convenient local approximation for the true worst-case risk. We follow in viewing adversarial examples more broadly, as "inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake." The little prior work studying adversarial examples in RL has assumed an p -norm threat model. and showed that deep RL policies are vulnerable to small perturbations in image observations. Recent work by generates a sequence of perturbations guiding the victim to a target state. Our work differs from these previous approaches by using a physically realistic threat model that disallows direct modification of the victim's observations. showed agents may become tightly coupled to the agents they were trained with. Like adversarial policies, this in seemingly strong polices failing against new opponents. However, the victims we attack win against a range of opponents, and so are not coupled in this way. Adversarial training is a common defense to adversarial examples, achieving state-of-the-art robustness in image classification . Prior work has also applied adversarial training to improve the robustness of deep RL policies, where the adversary exerts a force vector on the victim or varies dynamics parameters such as friction (; ;). Our defence of fine-tuning the victim against the adversary is inspired by this work. This work follows a rich tradition of worst-case analysis in RL. In robust MDPs, the transition function is chosen adversarially from an uncertainty set . solve the converse problem: finding the set of transition functions for which a policy is optimal. Methods also exist to verify controllers or find a counterexample to a specification. verify decision trees distilled from RL policies, while test black-box closedloop simulations. can even synthesise controllers robust to adversarial disturbances. Unfortunately, these techniques are only practical in simple environments with lowdimensional adversarial disturbances. By contrast, while our method lacks formal guarantees, it can test policies in complex multi-agent tasks and naturally scales with improvements in RL algorithms. We model the victim as playing against an opponent in a two-player Markov game . Our threat model assumes the attacker can control the opponent, in which case we call the opponent an adversary. We denote the adversary and victim by subscript α and ν respectively. The game M = (S, (A α, A ν), T, (R α, R ν)) consists of state set S, action sets A α and A ν, and a joint state transition function T: S × A α × A ν → ∆ (S) where ∆ (S) is a probability distribution on S. The reward function R i: S × A α × A ν × S → R for player i ∈ {α, ν} depends on the current state, next state and both player's actions. Each player wishes to maximize their (discounted) sum of rewards. The adversary is allowed unlimited black-box access to actions sampled from π v, but is not given any white-box information such as weights or activations. We further assume the victim agent follows a fixed stochastic policy π v, corresponding to the common case of a pre-trained model deployed with static weights. Note that in safety critical systems, where attacks like these would be most concerning, it is standard practice to validate a model and then freeze it, so as to ensure that the deployed model does not develop any new issues due to retraining. Therefore, a fixed victim is a realistic reflection of what we might see with RL-trained policies in real-world settings, such as with autonomous vehicles. Since the victim policy π ν is held fixed, the two-player Markov game M reduces to a single-player MDP M α = (S, A α, T α, R α) that the attacker must solve. The state and action space of the adversary are the same as in M, while the transition and reward function have the victim policy π ν embedded: where the victim's action is sampled from the stochastic policy a ν ∼ π ν (· | s). The goal of the attacker is to find an adversarial policy π α maximizing the sum of discounted rewards: Note the MDP's dynamics T α will be unknown even if the Markov game's dynamics T are known since the victim policy π ν is a black-box. Consequently, the attacker must solve an RL problem. We demonstrate the existence of adversarial policies in zero-sum simulated robotics games. First, we describe how the victim policies were trained and the environments they operate in. Subsequently, we provide details of our attack method in these environments, and describe several baselines. Finally, we present a quantitative and qualitative evaluation of the adversarial policies and baseline opponents. We attack victim policies for the zero-sum simulated robotics games created by Bansal et al. (2018a), illustrated in Figure 2. The victims were trained in pairs via self-play against random old versions of their opponent, for between 680 and 1360 million time steps. We use the pre-trained policy weights released in the "agent zoo" of Bansal et al. (2018b). In symmetric environments, the zoo agents are labeled ZooN where N is a random seed. In asymmetric environments, they are labeled ZooVN and ZooON representing the Victim and Opponent agents. All environments are two-player games in the MuJoCo robotics simulator. Both agents observe the position, velocity and contact forces of joints in their body, and the position of their opponent's joints. The episodes end when a win condition is triggered, or after a time limit, in which case the agents draw. We evaluate in all environments from Bansal et al. (2018a) except for Run to Goal, which we omit as the setup is identical to You Shall Not Pass except for the win condition. We describe the environments below, and specify the number of zoo agents and their type (MLP or LSTM): Sumo Ants (4, LSTM). The same task as Sumo Humans, but with'Ant' quadrupedal robot bodies. We use this task in Section 5.2 to investigate the importance of dimensionality to this attack method. Following the RL formulation in Section 3, we train an adversarial policy to maximize Equation 1 using Proximal Policy Optimization (PPO) . We give a sparse reward at the end of the episode, positive when the adversary wins the game and negative when it loses or ties. Bansal et al. (2018a) trained the victim policies using a similar reward, with an additional dense component at the start of training. We train for 20 million time steps using Stable Baselines's PPO implementation . The hyperparameters were selected through a combination of manual tuning and a random search of 100 samples; see Section A in the appendix for details. We compare our methods to three baselines: a policy Rand taking random actions; a lifeless policy Zero that exerts zero control; and all pre-trained policies Zoo * from Bansal et al. (2018a). We find the adversarial policies reliably win against most victim policies, and outperform the pre-trained Zoo baseline for a majority of environments and victims. We report Key: The solid line shows the median win rate for Adv across 5 random seeds, with the shaded region representing the minimum and maximum. The win rate is smoothed with a rolling average over 100, 000 timesteps. Baselines are shown as horizontal dashed lines. Agents Rand and Zero take random and zero actions respectively. The Zoo baseline is whichever ZooM (Sumo) or ZooOM (other environments) agent achieves the highest win rate. The victim is ZooN (Sumo) or ZooVN (other environments), where N is given in the title above each figure. the win rate over time against the median victim in each environment in Figure 3, with full in Figure 6 in the supplementary material. Win rates against all victims are summarized in Figure 4. Qualitative Evaluation The adversarial policies beat the victim not by performing the intended task (e.g. blocking a goal), but rather by exploiting weaknesses in the victim's policy. This effect is best seen by watching the videos at https://attackingrl.github.io/. In Kick and Defend and You Shall Not Pass, the adversarial policy never stands up. The adversary instead wins by taking actions that induce adversarial observations causing the victim's policy to take poor actions. A robust victim could easily win, a we demonstrate in Section 5.1. This flavor of attacks is impossible in Sumo Humans, since the adversarial policy immediately loses if it falls over. Faced with this control constraint, the adversarial policy learns a more high-level strategy: it kneels in the center in a stable position. Surprisingly, this is very effective against victim 1, which in 88% of cases falls over attempting to tackle the adversary. However, it proves less effective against victims 2 and 3, achieving only a 62% and 45% win rate, below Zoo baselines. We further explore the importance of the number of dimensions the adversary can safely manipulate in Section 5.2. Distribution Shift One might wonder if the adversarial policies are winning simply because they are outside the training distribution of the victim. To test this, we evaluate victims against two simple off-distribution baselines: a random policy Rand (green) and a lifeless policy Zero (red). These baselines win as often as 30% to 50% in Kick and Defend, but less than 1% of the time in Sumo and You Shall Not Pass. This is well below the performance of our adversarial policies. We conclude that most victim policies are robust to typical off-distribution observations. Although our adversarial policies do produce off-distribution observations, this is insufficient to explain their performance. In the previous section we demonstrated adversarial policies exist for victims in a range of competitive simulated robotics environments. In this section, we focus on understanding why these policies exist. Specifically, we establish that adversarial policies manipulate the victim through their body position; that victims are more vulnerable to adversarial policies in high-dimensional environments; and that activations of the victim's policy network differ substantially when playing an adversarial opponent. Adv is the best adversary trained against the victim, and Rand is a policy taking random actions. Zoo * N corresponds to ZooN (Sumo) or ZooON (otherwise). Zoo * 1T and Zoo * 1V are the train and validation datasets, drawn from Zoo1 (Sumo) or ZooO1 (otherwise). We have previously shown that adversarial policies are able to reliably win against victims. In this section, we demonstrate that they win by taking actions to induce natural observations that are adversarial to the victim, and not by physically interfering with the victim. To test this, we introduce a'masked' victim (labeled ZooMN or ZooMVN) that is the same as the normal victim ZooN or ZooVN, except the observation of the adversary's position is set to a static value corresponding to a typical initial position. We use the same adversarial policy against the normal and masked victim. One would expect it to be beneficial to be able to see your opponent. Indeed, the masked victims do worse than a normal victim when playing normal opponents. For example, Figure 4b shows that in You Shall Not Pass the normal opponent ZooO1 wins 78% of the time against the masked victim ZooMV1 but only 47% of the time against the normal victim ZooV1. However, the relationship is reversed when playing an adversary. The normal victim ZooV1 loses 86% of the time to adversary Adv1 whereas the masked victim ZooMV1 wins 99% of the time. This pattern is particularly clear in You Shall Not Pass, but the trend is similar in other environments. This is surprising as it implies highly non-transitive relationships may exist between policies even in games that seem to be transitive. A game is said to be transitive if policies can be ranked such that higher-ranked policies beat lower-ranked policies. Prima facie, the games in this paper seem transitive: professional human soccer players and sumo wrestlers can reliably beat amateurs. Despite this, there is a non-transitive relationship between adversarial policies, victims and masked victims. Consequently, we urge caution when using methods such as self-play that assume transitivity, and would recommend more general methods where practical . Our findings also suggest a trade-off in the size of the observation space. In benign environments, allowing more observation of the environment increases performance. However, this also makes the agent more vulnerable to adversaries. This is in contrast to an idealized Bayesian agent, where the value of information is always non-negative . In the following section, we investigate further the connection between vulnerability to attack and the size of the observation space. It is known that classifiers are more vulnerable to adversarial examples on high-dimensional inputs (b; ;). We hypothesize a similar for RL policies: the greater the dimensionality of the component P of the observation space under control of the adversary, the more vulnerable the victim is to attack. We test this hypothesis in the Sumo environment, varying whether the agents are Ants or Humanoids. The in Figures 4c and 4d support the hypothesis. The adversary has a much lower win-rate in the low-dimensional Sumo Ants (dim P = 15) environment than in the higher dimensional Sumo Humans (dim P = 24) environment, where P is the position of the adversary's joints. In Section 5.1 we showed that adversarial policies win by creating natural observations that are adversarial to the victim. In this section, we seek to better understand why these observations are adversarial. We record activations from each victim's policy network playing a range of opponents, and analyse these using a Gaussian Mixture Model (GMM) and a t-SNE representation. See Section B in the supplementary material for details of training and hyperparameters. We fit a GMM on activations Zoo * 1T collected playing against a normal opponent, Zoo1 or ZooV1, holding out Zoo * 1V for validation. Figure 5a shows that the adversarial policy Adv induces activations with the lowest log-likelihood, with random baseline Rand only slightly more probable. Normal opponents Zoo * 2 and Zoo * 3 induce activations with almost as high likelihood as the validation set Zoo * 1V, except in Sumo Humans where they are as unlikely as Rand. We plot a t-SNE visualization of the activations of Kick and Defend victim ZooV2 in Figure 5b. As expected from the density model , there is a clear separation between between Adv, Rand and the normal opponent ZooO2. Intriguingly, Adv induces activations more widely dispersed than the random policy Rand, which in turn are more widely dispersed than ZooO2. We report on the full set of victim policies in Figures 8 and 9 in the supplementary material. The ease with which policies can be attacked highlights the need for effective defences. A natural defence is to fine-tune the victim zoo policy against an adversary, which we term single training. We also investigate dual training, randomly picking either an adversary or a zoo policy at the start of each episode. The training procedure is otherwise the same as for adversaries, described in Section 4.2. We report on the win rates in You Shall Not Pass in Figure 4b. We find both the single ZooSV1 and dual ZooDV1 fine-tuned victims are robust to adversary Adv1, with the adversary win rate dropping from 87% to around 10%. However, ZooSV1 catastrophically forgots how to play against the normal opponent ZooO1. The dual fine-tuned victim ZooDV1 fares better, but still only wins 57% of the time against ZooO1, compared to 48% before fine-tuning. This suggests ZooV1 may use features that are helpful against a normal opponent but which are easily manipulable . Although the fine-tuned victims are robust to the original adversarial policy Adv1, they are still vulnerable to our attack method. New adversaries AdvS1 and AdvD1 trained against ZooSV1 and ZooDV1 win at equal or greater rates than before, and transfer successfully to the original victim. However, the new adversaries AdvS1 and AdvD1 are qualitatively different, tripping the victim up by lying prone on the ground, whereas Adv1 causes ZooV1 to fall without ever touching it. Contributions. Our paper makes three key contributions. First, we have proposed a novel threat model of natural adversarial observations produced by an adversarial policy taking actions in a shared environment. Second, we demonstrate that adversarial policies exist in a range of zero-sum simulated robotics games against state-of-the-art victims trained via self-play to be robust to adversaries. Third, we verify the adversarial policies win by confusing the victim, not by learning a generally strong policy. Specifically, we find the adversary induces highly off-distribution activations in the victim, and that victim performance increases when it is blind to the adversary's position. While it may at first appear unsurprising that a policy trained as an adversary against another RL policy would be able to exploit it, we believe that this observation is highly significant. The policies we have attacked were explicitly trained via self-play to be robust. Although it is known that self-play with deep RL may not converge, or converge only to a local rather than global Nash, self-play has been used with great success in a number of works focused on playing adversarial games directly against humans ). Our work shows that even apparently strong self-play policies can harbor serious but hard to find failure modes, demonstrating these theoretical limitations are practically relevant and highlighting the need for careful testing. Our attack provides some amount of testing by constructively lower-bounding the exploitability of a victim policy -its performance against its worst-case opponent -by training an adversary. Since the victim's win rate declines against our adversarial policy, we can confirm that the victim and its self-play opponent were not in a global Nash. Notably we expect our attack to succeed even for policies in a local Nash, as the adversary is trained starting from a random point that is likely outside the victim's attractive basin. Defence. We implemented a simple defence: fine-tuning the victim against the adversary. We find our attack can be successfully reapplied to beat this defence, suggesting adversarial policies are difficult to eliminate. However, the defence does appear to protect against attacks that rely on confusing the victim: the new adversarial policy is forced to instead trip the victim up. We therefore believe that scaling up this defence is a promising direction for future work. In particular, we envisage a variant of population-based training where new agents are continually added to the pool to promote diversity, and agents train against a fixed opponent for a prolonged period of time to avoid local equilibria. Table 1 gives the hyperparameters used for training. The number of environments was chosen for performance reasons after observing diminishing returns from using more than 8 parallel environments. The batch size, mini-batches, epochs per update, entropy coefficient and learning rate were tuned via a random search with 100 samples on two environments, Kick and Defend and Sumo Humans. The total time steps was chosen by inspection after observing diminishing returns to additional training. All other hyperparameters are the defaults in the PPO2 implementation in Stable Baselines . We repeated the hyperparameter sweep for fine-tuning victim policies for the defence experiments, but obtained similar . For simplicity, we therefore chose to use the same hyperparameters throughout. We used a mixture of in-house and cloud infrastructure to perform these experiments. It takes around 8 hours to train an adversary for a single victim using 4 cores of an Intel Xeon Platinum 8000 (Skylake) processor. We collect activations from all feed forward layers of the victim's policy network. This gives two 64-length vectors, which we concatenate into a single 128-dimension vector for analysis with a Gaussian Mixture Model and a t-SNE representation. We fit models with perplexity 5, 10, 20, 50, 75, 100, 250 and 1000. We chose 250 since qualitatively it produced the clearest visualization of data with a moderate number of distinct clusters. We fit models with 5, 10, 20, 40 and 80 components with a full (unrestricted) and diagonal covariance matrix. We used the Bayesian Information Criterion (BIC) and average log-likelihood on a heldout validation set as criteria for selecting hyperparameters. We found 20 components with a full covariance matrix achieved the lowest BIC and highest validation log-likelihood in the majority of environment-victim pairs, and was the runner-up in the remainder. Supplementary figures are provided on the subsequent pages. Figure 9: t-SNE activations of victim Zoo1 (Sumo) or ZooV1 (other environments). The are the same as in Figure 8 but decomposed into individual opponents for clarity. Model fitted with a perplexity of 250 to activations from 5000 timesteps against each opponent. Opponent Adv is the best adversary trained against the victim. Opponent Zoo is Zoo1 (Sumo) or ZooO1 (other environments). See Figure 8 for for other victims (one plot per victim).

Represent the TLDR for finding the corresponding scientific article

Deep RL policies can be attacked by other agents taking actions so as to create natural observations that are adversarial.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that a relatively simple black-box adversarial attack scheme using Bayesian optimization and dimension upsampling  is preferable to existing methods when the number of available queries is very low.