Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: Minecraft is a videogame that offers many interesting challenges for AI systems. In this paper, we focus in construction scenarios where an agent must build a complex structure made of individual blocks. As higher-level objects are formed of lower-level objects, the construction can naturally be modelled as a hierarchical task network. We model a house-construction scenario in classical and HTN planning and compare the advantages and disadvantages of both kinds of models. Minecraft is an open-world computer game, which poses interesting challenges for Artificial Intelligence BID0 BID12, for example for the evaluation of reinforcement learning techniques BID21. Previous research on planning in Minecraft focused on models to control an agent in the Minecraft world. Some examples include learning planning models from a textual description of the actions available to the agent and their preconditions and effects BID4, or HTN models from observing players' actions BID15., on the other hand, focused on online goal-reasoning for an agent that has to navigate in the minecraft environment to collect resources and/or craft objects. They introduced several propositional, numeric BID7 and hybrid PDDL+ planning models BID8.In contrast, we are interested in construction scenarios, where we generate instructions for making a given structure (e.g. a house) that is composed of atomic blocks. Our longterm goal is to design a natural-language system that is able to give instructions to a human user tasked with completing that construction. As a first step, in the present paper we consider planning methods coming up with what we call a construction plan, specifying the sequence of construction steps without taking into account the natural-language and dialogue parts of the problem. For the purpose of construction planning, the Minecraft world can be understood as a Blocksworld domain with a 3D environment. Blocks can be placed at any position having a non-empty adjacent position. However, while obtaining a sequence of "put-block" actions can be sufficient for an AI agent, communicating the plan to a human user requires more structure in order to formulate higher-level instructions like build-row, or build-wall. The objects being constructed (e.g. rows, walls, or an entire house) are naturally organized in a hierarchy where high-level objects are composed of lower-level objects. Therefore, the task of constructing a high-level object naturally translates into a hierarchical planning network (HTN) BID19 BID20 BID22 BID6.We devise several models in both classical PDDL planning BID5 BID13 ) and hierarchical planning for a simple scenario where a house must be constructed. Our first baseline is a classical planning model that ignores the high-level objects and simply outputs a sequence of place-blocks actions. This is insufficient for our purposes since the ing sequence of actions can hardly be described in natural language. However, it is a useful baseline to compare the other models. We also devise a second classical planning model, where the construction of high-level objects is encoded via auxiliary actions. HTN planning, on the other hand, allows to model the object hierarchy in a straightforward way, where there is a task for building each type of high-level object. The task of constructing each high-level object can be decomposed into tasks that construct its individual parts. Unlike in classical planning, where the PDDL language is supported by most/all planners, HTN planners have their own input language. Therefore, we consider specific models for two individual HTN planners: the PANDA planning system BID3 BID2 and SHOP2 BID14. We consider a simple scenario where our agent must construct a house in Minecraft. We model the Minecraft environment as a 3D grid, where each location is either empty or has a block of a number of types: wood, stone, or dirt. FIG0 shows the hierarchy of objects of our construction scenario. For the high-level structure the house consists of four stone walls, a stone roof, and a door. The walls and the roof are further decomposed into single rows that need to be built out of individual blocks. The door consists of two gaps, i.e., empty positions inside one of the walls. As our focus is on the construction elements we abstract low-level details away. For example, we avoid encoding the position of the agent and assume that all positions are always reachable. We also assume Minecraft's creative mode, where all block types are always available so we do not need to keep track of which blocks are there in the inventory. This is a very simplistic model, where planning focuses simply on the construction actions (i.e. placing or removing blocks), of high-level structures. Nevertheless, it can still pose some challenges to modern planners, specially due to the huge size of the Minecraft environment. Our first model is a classical planning model in the PDDL language that consists of only two actions: putblock(?location, ?block-type) and remove-block(?location, ? block-type) where there is a different location for each of the x-y-z coordinates in a 3D grid. The goal specifies what block-type should be in each location. As blocks cannot be placed in the air, the precondition of put-block requires one of the adjacent locations of? location to be non-empty. Other than that, blocks of any type can always be added or removed at any location. The goal is simply a set of block at facts. A limitation of this simple model is that it completely ignores the high-level structure of the objects being constructed. As there is no incentive to place blocks in certain order, a high-level explanation of the plan may be impossible. To address this, we introduce auxiliary actions that represent the construction of high-level objects. Figure 2 shows the auxiliary actions that represent building a wall. The attributes of the wall are specified in the initial state via attributes expressed by predicates wall dir, wall length, wall height, wall type, and current wall loc. In order to avoid the huge amount of combinations of walls that could be constructed of any dimensions and in any direction, the walls that are relevant for the construction at hand are specified in the initial state via these predicates. These three actions decompose the construction of a wall into several rows. Action begin wall ensures that no other high-level object is being constructed at the moment and adds the fact constructing wall to forbid the construction of any other wall (or roof) until the current wall has been finished. Action build row in wall ensures that a row of the given length will be built on the corresponding location and direction by adding predicates (building row) and (rest row ?loc ? len ?dir ?t). Simultaneously, it updates the location for the rest of the wall to be built and decreases its height by one. (not (constructing wall)) (not (current wall ? w))))Figure 2: Auxiliary PDDL actions to build a wall. When the height is zero, the action end wall becomes applicable, which finishes the construction of the wall. In the goal we then use the predicates wall at and roof at that force the planner to use these constructions, instead of a set of block at facts as we did in the simple model. HTN models encode the construction of high-level objects in a straightforward way by defining tasks such as build house, build wall and build row. These tasks will then be decomposed with methods until only primitive tasks will be left, in our case place-block and remove-block. We consider specific models for two individual HTN planners: the PANDA planning system BID3 BID2 ) and SHOP2 BID14. PANDA uses an HTN formalism BID9, which allows combining classical and HTN planning. The predicates describing the world itself, i.e. the relations between different locations remain the same as in the PDDL model, as do the place-block and remove-block primitive actions. On top of this, high-level objects are described as an HTN where each object corresponds to a task, without requiring to express their attributes with special predicates as we did in the PDDL model. Specifically, we defined tasks that correspond to building a house, a wall, a roof, a row of blocks, and the door. FIG1 shows the methods used to decompose the task of building a wall. These methods work in a recursive fashion over the height of the wall. For walls with height one, the build wall 1 method is used to build them. For walls with larger height, the build wall 2 method decomposes the task of building them into building a row in the current location and building the rest of the wall (i.e., a wall of height-1) in the location above the previous one. These subtasks are ordered, so that walls are always built from bottom to top. The methods for buildrow and buildroof work in the same fashion, while buildhouse only has one method decomposing the house into four walls, the roof, and the door. The task builddoor also has just one method stating which two blocks have to be removed to form a door. Choosing this way of modeling the door by first forcing the planner to place two blocks and later removing them again may seem inefficient, but for communication with a human user this may be preferable over indicating that these positions should remain empty in the first place. The SHOP2 model follows a similar hierarchical task structure as the PANDA model, having methods for decomposing the house into walls, a wall into rows and rows into single blocks. Since one of the advantages of SHOP2 is that it can call arbitrary LISP functions, we can represent the locations using integers as coordinates and replace the predicates used in PANDA and PDDL to express their relations by simple arithmetic operations. This also allows us to compute the end point of rows of any given length in a given direction, which means we can construct the walls by alternating the direction of the rows. Based on this, we define two different recursive decompositions of walls as shown in FIG2. In the first method we simply build the row starting in the current location, while in the second method we change the direction of the row we want to build and identify the position that would previously have been the end of the row by replacing the x-coordinate with x`length´1. Since this computation is different for each direction, we need separate methods for them. Apart from this, the decomposition structure is the same as with PANDA, building the walls, roof, and rows incrementally using a recursive structure. To evaluate the performance of common planners on our models 1, we scale them with respect to two orthogonal parameters: the size of the construction, and the size of the cubic 3D world we are considering. We use different planners for each model. For the classical planning models we use the LAMA planner BID16. The PANDA planning system implements several algorithms, including plan space POCL-based search methods BID3 BID2, SAT-based approaches, and forward heuristic search. We use a configuration using heuristic search with the FF heuristic, which works well on our models. For SHOP2, we use the depthfirst search configuration BID14. All experiments were run on an Intel i5 4200U processor with a time limit of 30 minutes and a memory limit of 2GB.In our first experiment, we scale the size of the house starting with a 3ˆ3ˆ3 house and increasing one parameter (length, width, and height) at a time (4ˆ3ˆ3, 4ˆ41 3, . . ., 9ˆ9ˆ9.). The size of the 3D world is kept as small as possible to fit the house with some slack, so initially is set to 5ˆ5ˆ5 and is increased by one unit in each direction every three steps, once we have scaled the house in all dimensions. The upper row of FIG3 shows the search and total time of the planners on the different models. The construction size in the x-axis refers to the number of blocks that need to be placed in the construction. All planners scale well with respect to search time, solving problems of size up to 9ˆ9ˆ9 in just a few seconds. The non-hierarchical PDDL planning model (PDDL blocks) that only uses the place-block and remove-block actions without any hierarchical information is the one with worst search performance. Moreover, it also in typically longer plans that build many "support" structures to place a block in a wall without one of the adjacent blocks in the wall being there yet. However, there is a huge gap between search and total time for the PANDA and PDDL models, mostly due to the overhead of the grounding phase. SHOP2 does not do any preprocessing or grounding so it is not impacted by this. For the PANDA and PDDL models, total time significantly increases every three problems, whenever the world size is increased. This suggests that, somewhat counterintuitively, the size of the world environment has a greater impact on these planners' performance than the size of the construction. In the PDDL based approaches, the number of operators and facts produced in the preprocessing shows a similar trend so the planner's performance seems directly influenced by the size of the grounded task. For PANDA, on the other hand, we observe a linear increase in the number of facts and only a comparatively small increase in the number of operators. To test more precisely what is the impact of increasing the world size, we ran a second set of experiments where we kept the size of the house fixed at 5ˆ5ˆ5 and just increased the size of the world. As shown in the bottom part of FIG3 the performance of SHOP2 is not affected at all, since it does not require enumerating all possible locations. Search time for PANDA also stays mostly constant, but the overhead in the preprocessing phase dominates the total time. This contrasts with the number of operators and facts, which is not affected by the world size at all. The PDDL based models are also affected in terms of preprocessing time, due to a linear increase in the number of facts and operators with respect to world size, but to a lesser degree. However, search time increases linearly with respect to the world size due to the overhead caused in the heuristic evaluation. We have introduced several models of a construction scenario in the Minecraft game. Our experiments have shown that, even in the simplest construction scenario which is not too challenging from the point of view of the search, current planners may struggle when the size of the world increases. This is a serious limitation in the Minecraft domain, where worlds with millions of blocks are not unrealistic. Lifted planners like SHOP2 perform well. However, it must be noted that they follow a very simple search strategy, which is very effective on our models where any method decomposition always leads to a valid solution. However, it may be less effective when other constraints must be met and/or optimizing quality is required. For example, if some blocks are removed from the ground by the user, then some additional blocks must be placed as auxiliary structure for the main construction. Arguably, this could be easily fixed by changing the model so that whenever a block cannot be placed in a target location, an auxiliary tower of blocks is built beneath the location. However, this increases the burden of writing new scenarios since suitable task decompositions (along with good criteria of when to select each decomposition) have to be designed for all possible situations. This makes the SHOP2 model less robust to unexpected situations that were not anticipated by the domain modeler. PANDA, on the other hand, supports insertion of primitive actions BID9, allowing the planner to consider placing additional blocks, e.g., to build supporting structures that do not correspond to any task in the HTN. This could help to increase the robustness of the planner in unexpected situations where auxiliary structures that have not been anticipated by the modeler are needed. However, this is currently only supported by the POCL-plan-based search component and considering all possibilities for task insertion significantly slows down the search and it runs out of memory in our scenarios. This may point out new avenues of research on more efficient ways to consider task insertion. In related Minecraft applications, cognitive priming has been suggested as a possible solution to keep the size of the world considered by the planner at bay BID17. In construction scenarios, however, large parts of the environment can be relevant so incremental grounding approaches may be needed to consider different parts of the scenario at different points in the construction plan. Our models are still a simple prototype and they do not yet capture the whole complexity of the domain. We plan to extend them in different directions in order to capture how hard it is to describe actions or method decompositions in natural language. For example, while considering the position of the user is not strictly necessary, his visibility may be important because objects in his field of view are easier to describe in natural language. How to effectively model the field of vision is a challenging topic, which may lead to combinations with external solvers like in the planning modulo theories paradigm BID10.Another interesting extension is to consider how easy it is to express the given action in natural language and for example by reducing action cost for placing blocks near objects that can be easily referred to. Such objects could be landmarks e.g. blocks of a different type ("put a stone block next to the blue block") or just the previously placed block (e.g., "Now, put another stone block on top of it").

We model a house-construction scenario in Minecraft in classical and HTN planning and compare the advantages and disadvantages of both kinds of models.

This paper introduces partial grounding to tackle the problem that arises when the full grounding process, i.e., the translation of a PDDL input task into a ground representation like STRIPS, is infeasible due to memory or time constraints.


Question: Represent the article for finding a single-sentence TLDR

Question: Attacks on natural language models are difficult to compare due to their different definitions of what constitutes a successful attack. We present a taxonomy of constraints to categorize these attacks. For each constraint, we present a real-world use case and a way to measure how well generated samples enforce the constraint. We then employ our framework to evaluate two state-of-the art attacks which fool models with synonym substitution. These attacks claim their adversarial perturbations preserve the semantics and syntactical correctness of the inputs, but our analysis shows these constraints are not strongly enforced. For a significant portion of these adversarial examples, a grammar checker detects an increase in errors. Additionally, human studies indicate that many of these adversarial examples diverge in semantic meaning from the input or do not appear to be human-written. Finally, we highlight the need for standardized evaluation of attacks that share constraints. Without shared evaluation metrics, it is up to researchers to set thresholds that determine the trade-off between attack quality and attack success. We recommend well-designed human studies to determine the best threshold to approximate human judgement. Advances in deep learning have led to impressive performance on many tasks, but models still make mistakes. Models are particularly vulernable to adversarial examples, inputs designed to fool models . demonstrated that image classification models could be fooled by perturbations indistinguishable to humans. Due to the importance of natural language processing (NLP) tasks, a large body of research has focused on applying the concept of adversarial examples to text, including (; ; ; ; ; ; ; ; ; ; a). The importance of tasks such as spam and plagiarism detection highlights the need for robust NLP models. However, there are fundamental differences between image and text data. Unlike images, two different sequences of text are never entirely indistinguishable. This raises the question: if indistinguishable perturbations aren't possible, what are adversarial examples in text? We observe that each work from recent literature has a slightly different definition of what constitutes an adversarial example in natural language. Comparing the success rate of two attacks is meaningless if the attacks use different methods to evaluate the same constraints or define different constraints altogether. In this paper, we build on to introduce a taxonomy of constraints specific to adversarial examples in natural language. To the best of our knowledge, our work provides the first comprehensive framework for categorizing and evaluating attack constraints in natural language. We discuss use cases and propose standardized evaluation methods for each of these constraints. We then apply our evaluation methods to the synonym-substitution based attacks of and. These attacks claimed to preserve the syntax and semantics of the original sentence, while remaining non-suspicious to a human interpreter. However, we find that most of their adversarial examples contain additional grammatical errors, and human surveys reveal that many adversarial examples also change the meaning of the sentence and/or do not appear to be written by humans. These call into question the ubiquity of synonym-based adversarial examples and emphasize the need for more careful evaluation of attack approaches. Lastly, we discuss how previous works rely on arbitrary thresholds to determine the semantic similarity of two sentences. These thresholds can be tuned by the researcher to make their methods seem more successful with little penalty in quantitative metrics. Thus, we highlight the importance of standardized human evaluations to approximate the true threshold value. Any method that introduces a novel approach to measure semantic similarity should support their choice of threshold with defensible human studies. The three main contributions of this paper are: • We formally define and categorize constraints on adversarial examples in text, and introduce evaluation methods for each category. • Using these categorizations and evaluation methods, we quantitatively disprove claims that stateof-the-art synonym-based substitutions preserve semantics and grammatical correctness. • We show the sensitivity of attack success rate to changes in semantic similarity thresholds set by researchers. We assert that perturbations which claim semantic similarity should use standardized human evaluation studies with precise wording to determine an appropriate threshold. Research has shown that current deep neural network models lack the ability to classify adversarial examples correctly . Two images that appear identical to a human but are different by a minuscule amount can be assigned grossly different classification scores by a model. These adversarial examples expose issues that arise from the black-box nature of deep learning predictions. Studying them is key to eventually building secure applications based on deep learning. The search for adversarial examples can be framed as a simple optimization problem: maximizing the change in prediction while minimizing the change in input (; ; b;). Past work in the image domain explored applying a small factor of the gradient to move an input in the "worst-case" direction. applied a similar approach for text by mapping sentences into latent space. Since then, most approaches have been more heuristic-driven due to the difficulty of maintaining syntactic and semantic similarity otherwise. We categorize these attacks into four groups. Attacks by Character Substitution: Recently, multiple studies proposed to attack natural language classification models by replacing words with deliberately misspelled words (; ;). These studies used character replacements to change a word into one that the model doesn't recognize. Such replacements are designed to create character sequences that a human would easily correct into the intended words. Character replacements, while effective in certain circumstances, can be detected with any spellchecking software. demonstrated a way to train a model that correctly classifies these misspelled inputs. Attacks by Word Insertion or Removal: and devised a way to determine the most important words in the input and then used heuristic-driven rules to generate perturbed inputs by adding or removing important words. appended extra sentences to paragraphs that were inputs to reading comprehension tasks. These three studies each point out interesting flaws in natural language models, but do not produce adversarial examples that retain the semantics of the original input. Attacks by Paraphrase: The bulk of recent work has defined adversarial examples in natural language as any sequences that both fool the model and share the meaning of the original input . Thus, if document x is of class y, then any paraphrase of x that is not of class y is a bona-fide adversarial example. and proposed to use neural machine translation systems to paraphrase input text to create semantic-preserving perturbations. However, these systems have difficulty generating diverse, syntactically correct paraphrases. Their paraphrases often have bad grammar and overall have trouble fooling the model. Attacks by Synonym Substitution: Due to the difficulty of generating paraphrases in general,;;; Papernot et al. (2016a) have developed easier ways to generate a subset of all paraphrases: by replacing a certain number of words from the input with synonyms. proposed generating word-level swaps with a population-based optimization algorithm. The study aimed to generate semantically and syntactically similar adversarial examples that fool well-trained sentiment analysis and textual entailment models. In order to select the best replacement word, the study computed the nearest neighbors of a selected word according to the distance in the counter-fitted embedding space. It also used the Google one billion words language model to filter out words that do not fit within the context. Another recent system, TextFooler, was proposed by to attack three DNN models, including the powerful pre-trained BERT , and the widely used convolutional and recurrent neural networks. The method identifies the most important words in the input sequence and replaces them one-by-one with synonyms until the model output changes. The study used counter-fitted word embeddings to select the best replacement word and then utilized the Universal Sentence Encoder to measure a similarity score between the perturbed and original inputs. and used a technique called Interval Bound Propagation (IBP) to train models that withstand synonym attacks. IBP encourages models to make the same prediction on all sequences within a predefined neighborhood. In this case, it ensures that sentences that are a few synonym swaps away from each other have similar prediction scores. However, IBP is currently only feasible for simple models: feedforward networks, CNNs, and LSTMs with only a few layers. Perhaps more problematic, training with IBP causes a substantial drop in accuracy on the test set. define a broad framework to determine if a machine learning model is secure, but do not specify to adversarial examples on text. laid out a set of potential constraints for the attack space when generating adversarial examples, which are each useful in different real-world scenarios. defined a framework for evaluating attacks on machine translation models, focusing on meaning preservation constraints, but restricted their definitions to sequence-to-sequence models. When constructing adversarial examples, it's important to define the space of inputs available to the attacker. A successful adversarial example is defined as one that fools the model, but other constraints are often defined to restrict the outputs that an attacker may produce. For a given classifier F and a test input x, we denote y gt as the ground-truth output, F (x) as the model output, and C 0, C 1,... C n as a set of functional constraints, each of which represents whether an attack satisfies a certain constraint. An example x adv is considered adversarial if: Adversarial examples aim to achieve two goals at the same time: to induce misclassification by achieving F (x adv) = y gt (we focus on untargeted attacks), and to restrict x adv. For instance, C 0 (x, x adv) requires morphology be preserved between x and x adv. Unlike with images, text perturbations are never indistinguishable. The attacker must decide what makes two natural language inputs indistinguishable based on the scenario. This increases the importance of clearly defining constraints. Different sets of constraints are appropriate for different use cases. We extend on the categorization of attack spaces for adversarial examples introduced by to introduce a taxonomy of constraints for natural language specifically, and discuss possible motivations for studying each constraint. We propose a method for evaluating whether adversarial examples meet each constraint. We define three mutually exclusive categories of semantics-related constraints on attacks in natural language: altering an input sequence while changing as few characters as possible (morphologicallypreserving), altering an input sequence while retaining its meaning (semantics-preserving), and crafting an input sequence which contains a specific message (semantics-constrained). We then define two constraints which may be used in addition to any semantic constraints: a requirement for adversarial examples to be grammatically correct, and the requirement to appear to have been written by a human (non-suspicious). Appendix A.1 categorizes a selection of prior work into these groups. 3.1 MORPHOLOGY-PRESERVING PERTURBATION In some situations, the attacker is willing to change the semantics of the input as long as all changed sentences read the same to a human. These character substitutions may change the semantics of the sentence, as long as the reader can figure out the intended words from the context. If no defense is provided for morphological perturbations, an attacker can easily transmit a message by adding or a deleting a handful of characters . A perturbation is morphology-preserving if satisfying C(x, x adv):= {d(x, x adv) ≤ }, where d measures the morphological distance between two strings. One common choice of d for morphological distance is d edit (x, x adv), the edit distance between x and x adv. Edit distance is the minimum number of operations (insertions, deletions, or substitutions) required to transform the string x adv back into x. 3.2 SEMANTICS-PRESERVING PERTURBATION The attacker starts with an input sequence and is allowed to modify the input sequence however they want as long as the semantics is preserved. A use case for this class of adversarial examples is tricking plagiarism detection software. An attacker wants to preserve as much of the original document as possible but still avoid detection. An adversarial example is semantics-preserving if a human labeler agrees that any changes between the input sequence x and the modified sequence x adv do not alter the meaning of the sequence. We define d sem (x, x adv) as the score given by humans when asked to rate if the meaning is preserved on a Likert scale of 1-5, where 1 is "Strongly Agree" and 5 is "Strongly Disagree" . A perturbation is semantics-preserving if: We propose = 2 as a general rule: on average, humans should either "agree" or "strongly agree" that x and x adv have the same meaning. Evaluation of semantics-preserving perturbations should ideally be done by humans, but machine evaluation of semantic similarity can be used as a proxy for human judgment, as explored by. Automated metrics of meaning should always be supported by human studies. The attacker may generate any input sequence as long as it contains the semantic content that the attacker intends. This is distinct from the semantics-preserving case because there is no starting input sequence to perturb; the attacker can convey their message by any means possible. In these cases, the attacker can write anything, as long as it conveys specific meaning. As discussed by , one use case for semantics-constrained input is to fool a spam classifier. The attacker finds some way to relay their message that evades spam detection. The attacker may create any sequence as long as it contains the semantic content they desire. Examples with this constraint are more difficult to generate than semantics-preserving perturbations because the search space is much bigger and it is challenging to check that the model produces incorrect output without human review. Evaluation of semantics-constrained input is more difficult than semantics-preserving perturbations, as there is no starting point to compare to. There will never be automated way to tell if an email is spam with 100% accuracy. Whether an input is semantics-constrained must be determined by a human judge. 3.4 SYNTACTIC CONSTRAINT Under this constraint, the attacker is constrained to inputs that are grammatically valid. Perturbations that introduce grammar errors often aren't semantics-preserving. However, in many cases, a human reader will recognize that x and x adv have the same meaning even if x adv introduced additional syntactic errors. There may be use cases for both semantics-preserving perturbations and semanticsconstrained inputs with and without syntactic constraints: • Semantics-preserving perturbation with syntactic constraint: Consider a student who wishes to alter a copied assignment to evade plagiarism detection. They must perturb the assignment while preserving the meaning, but introducing grammatical errors would get them a bad grade. • Semantics-preserving perturbation without syntactic constraint: An attacker who wishes to illegally distribute a PDF online must retain the content in the PDF, but readers of the PDF are indifferent to whether grammar errors are introduced as long as the meaning is unchanged. Similar to the case of illegal video streaming discussed in , the consumers of the PDF may be thought of as colluding with the attacker. • Semantics-constrained input with syntactic constraint: An adversary wishing to evade a fake news classifier must generate an article with correct grammar. • Semantics-constrained input without syntactic constraint: Someone wishing to post offensive content on social media may be willing to purposely misspell words or insert grammatical errors in order to get their content onto a platform. Evaluation of whether adversarial examples follow the grammatical constraint requires a way of measuring the amount of grammar errors in a document. Define G(s) as the number of syntactic errors in an input sequence s. When applying a semantics-preserving perturbation from x to x adv, the grammatical constraint may be represented as where represents the amount of additional grammatical errors the adversary is willing to introduce. When generating input from scratch, the constraint becomes simply: G(x adv) ≤, where is the amount of total grammatical errors the adversary is willing to create. Methods which claim to introduce no grammatical errors imply = 0. Evaluation of G(x) and G(x adv) may be done by humans, but pragmatically it is more convenient and consistent to use an automatic grammar checker such as LanguageTool . 3.5 NON-SUSPICIOUS CONSTRAINT The non-suspicious constraint specifies that the adversarial example must appear to be humanwritten to other humans. It has some overlap with the syntactic constraint, but they do not always go together. Sentences without grammar errors may still not seem human-written. Conversely, grammatically incorrect sentences could always have been written by a human. An example use case of an attack with the non-suspicious constraint is the plagiarism example discussed above: the student would not want to make modifications to their assignment such that the teacher would be able to tell the assignment was not human-written. A case where the non-suspicious constraint does not apply is the illegal PDF as mentioned earlier, as the consumers of the PDF do not care if it has been altered as long as meaning is preserved. Evaluation of the non-suspicious constraint must be done by humans. We propose a method in which humans are shown a shuffled mix of real and adversarial sequences and must guess whether each one is real or computer-altered, or computer-generated in the case of semantics-constrained input. Define the portion of people who correctly identify an adversarial example x adv as computeraltered as R(x adv) An adversarial example is considered to meet the non-suspicious constraint if R(x adv) <, where 0 < ≤ 1. A smaller value of enforces more rigorous standards on what must appear to be written by a human. We now apply the evaluation framework discussed in Section 3 to evaluate the effectiveness of the attack techniques from and. We chose these these works because: • They claim to find semantics-preserving perturbations which adhere to the syntactic constraint, and imply adherence to the non-suspicious constraint. However, our inspection of the adversarial perturbations revealed that many introduced syntax errors or did not preserve semantics. • They report high attack success rates 1. • These methods cover two of the most effective models for text classification: LSTM and BERT. To generate examples for evaluation, we attacked BERT using's method and attacked a WordLSTM using's method. We only considered classification tasks as our focus is on evaluating the adversarial examples. We evaluate both methods on the IMDB 2 and Yelp polarity document-level sentiment classification datasets and's method on the MR sentence-level sentiment classification dataset . We ran experiments to evaluate whether the generated examples met three constraints: syntax, semanticspreservation and non-suspicion. We evaluate fulfillment of the syntactic constraint using a grammar checker. These programs are easy to use, free, and cover a wide range of grammatical rules over many languages. We chose to use LanguageTool, an open-source proofreading tool . LanguageTool ships with thousands of human-curated rules for the English language and provides a downloadable server interface for analyzing sentences. We ran each of the generated (x, x adv) pairs through LanguageTool and compared the amount of detected errors to determine if the syntactic constraint as defined in Equation 3 was fulfilled. LanguageTool detected more grammatical errors in x adv for 52.1% of pairs across the five datasets. There is a clear linear relationship between the number of words changed and the number of grammatical errors induced (Figure 1). Clearly, the majority of generated examples don't fulfill the syntactic constraint. As shown in A savvy defender who wants to build a system robust against these types of substitution attacks may notice that adversarial examples contain strange phrasings and unnatural errors that a human would almost never make. LanguageTool detected errors in several categories that appeared far often in adversarial samples than in the original, human-generated content. As discussed in Section 3.2, tools for automatically evaluating semantic similarity exist, but human judgement is essential to confirm such automatic evaluations. and both include human evaluation, these studies include a single scale on which users rated the similarity of x and x adv to be relatively high. We believe that this phrasing did not generate an accurate measure of semantic similarity, as users may have considered two sentences with clearly different meanings to be "similar" due to their small morphological distance. Instead, we ask users to determine whether the changes between x and x adv preserve the meaning of the passage. We set up our own survey to evaluate human judgement on the adversarial examples from and. We enlisted workers from Amazon Mechanical Turk to label 3907 data points across the five datasets. Our survey randomized the order of x and x adv and displayed them side-by-side. To evaluate whether the perturbation was semantics-preserving as defined in Equation 2, we asked users to rate whether they agreed that the changes between the two passages preserved the meaning of the passage on a scale of 1 (Strongly Agree) to 5 (Strongly Disagree). Average scores as shown in Figure 2 vary by dataset, but users' answers generally generally average out to around 3 ("Not Sure"). Clarifying the survey question makes it clear that many of the examples are not semantics-preserving. Additionally, we note that due to acquiescence bias, labelers are generally more likely to agree than disagree when presented with a statement 3. To test this effect, we negated the question: instead of asking whether the changes from x to x adv "preserved" the meaning of x, we asked whether the differences "changed" the meaning. Table 3: Likert scale response: Does changing from x to x adv change meaning? Table 3 shows that averages are now closest to "2 -Agree" rather than "3 -Not sure". Thus, inverting the question ed in generated examples being rated as less semantics-preserving. Future human evaluation studies should use similar practices to minimize the effects of this bias. The final claim we investigated was whether adversarial examples generated by these two attack techniques were truly non-suspicious. We presented examples one by one to humans and asked if they were real or computer-altered as proposed in Section 3.5. A method that consistently generated non-suspicious adversarial examples would produce inputs would appear to be human-written. In this case, human labelers would average an accuracy of 50%. If humans could consistently discern perturbed inputs from real inputs, they would achieve 100% on this task. As this is a time-consuming task for long documents, we only evaluated adversarial examples generated by 's method on the sentence-level MR dataset. We sampled 100 (x, x adv) pairs then used Mechanical Turk to get 10 guesses for each example. Humans achieved 69.2% accuracy on this task. Table 4 presents the confusion matrix of from the survey. Interestingly, workers guessed that the examples were real 62.2% of the time, but when they guessed that examples were computer-altered they were right 75.4% of the time. Thus while some perturbed examples are non-suspicious, there are some which workers identify with high precision. Original 814 431 Perturbed 186 570 Sometimes, prior work has chosen the same metric d but different thresholds for. Synonym substitution attacks provide one example. These attacks consider two sentences (x, x adv) that differ by a single word (w ∈ x, w adv ∈ x adv) to have the same meaning if w adv is one of the closest neighbors in the counter-fitted embedding space. In other words, indicates how many synonyms they allowed. At least three different such attacks rely on this metric: Alzantot et al. uses = 8, Kuleshov et al. uses = 15, and Jin et al. (2019 uses = 50. Without an agreed-upon standard value, choice of is at the researcher's discretion. This is problematic because there is a direct correlation between the value chosen for and the success of an attack. used an additional distance metric d defined as the cosine similarity between embeddings encoded by the Universal Sentence Encoder (USE) in order to determine if a synonym swap preserves semantic similarity . Figure 3 shows accuracy under BERT under attack by's method as the maximum allowed cosine similarity between two sentences' USE embeddings increases 4. As becomes more strict, the attack becomes less successful. Figure 4 plots the accuracy under attack as the number of synonyms considered for each substitution increases. An attack that is more lenient with its standard for what constitutes a synonym is more successful. For any distance metric, there is no perfect value of. Whether two sentences are truly paraphrases is often subjective. We cannot expect any automated evaluation tool to perfectly separate paraphrases from non-paraphrases. However, there will be a range of values for that correlate the most closely with the judgement of humans. Without using standardized human studies like the one in Section 4.2 to verify the linguistic standard provided by the choice of, generated adversarial examples may not be useful at all. The correct range of values of will vary by metric. Every study that presents a new way to measure semantic similarity should perform a human study to approximate the proper value for. With consistent values for, researchers will be able to easily compare methods that use the same metrics. We introduced a framework for evaluating fulfillment of attack constraints in natural language. Applying this framework to synonym substitution attacks raised concerns about the semantic preservation, syntactic accuracy, and conspicuity of the adversarial examples they generate. Future work may expand our hierarchy to categorize and evaluate different attack constraints in natural language. Standardized terminology and evaluation metrics will make it easier for defenders to determine which attacks they must protect themselves from-and how. It remains to be seen how robust BERT is when subject to synonym attacks which rigorously preserve semantics and syntax. It is up to future research to determine how prevalent adversarial examples are throughout the broader space of paraphrases. A.1 CATEGORIZATION OF NLP ATTACKS Past research in attacks on textual models has provided many disparate sets of constraints for adversarial examples. Here, we select a small slice of the past work and categorize it based on the taxonomy outlined in Section 3. Constraint on Meaning Syntactic Constraint Non-suspicious Synonym Substitution. (; ;) semantics-preserving perturbation Character Substitution. (; ;) morphology-preserving perturbation Word Insertion or Removal. ) semantics-preserving perturbation General Paraphrase. (; ;) semantics-preserving perturbation Good Word Attack. semantics-preserving perturbation Sentence Insertion. semantics-preserving perturbation A.2 DETAILS ABOUT HUMAN STUDIES. Our experiments relied on labor crowd-sourced from Amazon Mechanical Turk to provide labels for two tasks. We used five datasets: MIT and Yelp datasets from and MIT, Yelp, and Movie Review datasets from . We added a limitation so that only workers with "Masters" status on Mechanical Turk could complete our tasks. We estimated that each task would take approximately 15 seconds to complete, so we paid workers $.05 per label to ensure a fair wage of $12 per hour. For both tasks, we allotted workers 3 minutes per assignment. Task 1: Real vs. fake? In Section 4.3, we present from our Mechanical Turk survey where we asked users to guess if a sample of text was real or Computer-altered. We restricted this task to a single dataset, Movie Review. We chose Movie Review because it had an average sample length of 20 words, much shorter than Yelp or IMDB. We made this restriction because of the time-consuming nature of classifying long samples as Real or Fake. Task 2: Semantic Similarity. In Section 4.2, we present from two Mechanical Turk questionnaires to judge semantic similarity or dissimilarity. For each task, we show x and x adv, in a random order. We added a custom script to highlight the different characters between the two sequences. For both tasks, we provided the following description: "Compare two short pieces of English text and determine if they mean different things or the same." We then prompted labelers: "Changing from one of these sentences to the other X the meaning," where X was "changes" or "preserves". Phrasing matters. Mechanical Turk comes with a set of pre-designed questionnaire interfaces. These include one titled "Semantic Similarity" which asks users to rate a pair of sentences on a scale from "Not Similar At All" to "Highly Similar." Examples generated by synonym attacks benefit from this question formulation because humans tend to rate two sentences that share many words as "Similar" due to their small morphological distance, even if they have different meanings. Cognitive biases in the Likert scale. It was interesting to compare from the surveys within Task 2. Since all we did was negate the question, the should have been inverses: i.e., every label that said "Agree" for the first formulation of the question should have said "Disagree" for the second, etc. However, we found that labelers, at least on the Mechanical Turk, tend to select "Agree" regardless of the question. When asked if the change from x to x adv "changes" meaning, 68.3% of labelers selected "Agree". Surprisingly, when asked if the change from x to x adv "preserves" meaning, 43.6% of labelers still answered with "Agree"! Overall, 55.9% of our 7,816 responses, from two opposite surveys, were simply "Agree". We believe that this is an instance of a phenomenon called acquiescence bias, where respondents to a survey gravitate towards answers that are generally positive . Notes for future surveys. In the future, we would try to filter out bad labels by mixing a small number of ground-truth "easy" data points into our dataset and rejecting the work of labelers who performed poorly on this set. A.3 LANGUAGETOOL ERROR TYPES LanguageTool presents helpful error message along with each detected grammatical rule violation. This table displays sample error messages alongside the error codes from Figure 2. For example, code PRP RB NO VB appears 20 times more frequently in adversarial samples than it does in real, human-crafted input. This specific error often appears when the attack substitutes a verb with a noun. 6 TO NON BASE 12.0 The verb after "to" should be in the base form:'constitute'. 7 PRP PAST PART 8.7 Possible grammatical error. You used a past participle without using any required verb ('be' or 'have'). Did you mean'was','were'? 8 AFFORD VB 7.5 This verb is used with the infinitive:'to better','to well'

We present a framework for evaluating adversarial examples in natural language processing and demonstrate that generated adversarial examples are often not semantics-preserving, syntactically correct, or non-suspicious.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.


Question: Represent the scientific article to find a short TLDR

Question: In order for machine learning to be deployed and trusted in many applications, it is crucial to be able to reliably explain why the machine learning algorithm makes certain predictions. For example, if an algorithm classifies a given pathology image to be a malignant tumor, then the doctor may need to know which parts of the image led the algorithm to this classification. How to interpret black-box predictors is thus an important and active area of research. A fundamental question is: how much can we trust the interpretation itself? In this paper, we show that interpretation of deep learning predictions is extremely fragile in the following sense: two perceptively indistinguishable inputs with the same predicted label can be assigned very different}interpretations. We systematically characterize the fragility of the interpretations generated by several widely-used feature-importance interpretation methods (saliency maps, integrated gradient, and DeepLIFT) on ImageNet and CIFAR-10. Our experiments show that even small random perturbation can change the feature importance and new systematic perturbations can lead to dramatically different interpretations without changing the label. We extend these to show that interpretations based on exemplars (e.g. influence functions) are similarly fragile. Our analysis of the geometry of the Hessian matrix gives insight on why fragility could be a fundamental challenge to the current interpretation approaches. Predictions made by machine learning algorithms play an important role in our everyday lives and can affect decisions in technology, medicine, and even the legal system . As the algorithms become increasingly complex, explanations for why an algorithm makes certain decisions are ever more crucial. For example, if an AI system predicts a given pathology image to be malignant, then the doctor would want to know what features in the image led the algorithm to this classification. Similarly, if an algorithm predicts an individual to be a credit risk, then the lender (and the borrower) might want to know why. Therefore having interpretations for why certain predictions are made is critical for establishing trust and transparency between the users and the algorithm .Having an interpretation is not enough, however. The explanation itself must be robust in order to establish human trust. Take the pathology predictor; an interpretation method might suggest that a particular section in an image is important for the malignant classification (e.g. that section could have high scores in saliency map). The clinician might then focus on that section for investigation, treatment or even look for similar features in other patients. It would be highly disconcerting if in an extremely similar image, visually indistinguishable from the original and also classified as malignant, a very different section is interpreted as being salient for the prediction. Thus, even if the predictor is robust (both images are correctly labeled as malignant), that the interpretation is fragile would still be highly problematic in deployment. Our contributions. The fragility of prediction in deep neural networks against adversarial attacks is an active area of research BID4;; ). In that setting, fragility is exhibited when two perceptively indistinguishable images are assigned different labels by the neural network. In this paper, we extend the definition of fragility to neural network interpretation. More precisely, we define the interpretation of neural network to be fragile if perceptively indistinguishable images that have the same prediction label by the neural network are given substantially different interpretations. We systematically The fragility of feature-importance maps. We generate feature-importance scores, also called saliency maps, using three popular interpretation methods: simple gradient (a), DeepLIFT (b) and integrated gradient (c). The top row shows the the original images and their saliency maps and the bottom row shows the perturbed images (using the center attack with = 8, as described in Section 3) and the corresponding saliency maps. In all three images, the predicted label has not changed due to perturbation; in fact the network's (SqueezeNet) confidence in the prediction has actually increased. However, the saliency maps of the perturbed images are meaningless.investigate two classes of interpretation methods: methods that assign importance scores to each feature (this includes simple gradient , DeepLift , and integrated gradient ), as well as a method that assigns importances to each training example: influence functions . For both classes of interpretations, we show that targeted perturbations can lead to dramatically different interpretations (FIG0).Our findings highlight the fragility of interpretations of neural networks, which has not been carefully considered in literature. Fragility directly limits how much we can trust and learn from the interpretations. It also raises a significant new security concern. Especially in medical or economic applications, users often take the interpretation of a prediction as containing causal insight ("this image is a malignant tumor likely because of the section with a high saliency score"). An adversary could minutely manipulate the input to draw attention away from relevant features or onto his/her desired features. Such attacks might be especially hard to detect as the actual labels have not changed. While we focus on image data here because most of the interpretation methods have been motivated by images, the fragility of neural network interpretation could be a much broader problem. Fig. 2 illustrates the intuition that when the decision boundary in the input feature space is complex, as is the case with deep nets, a small perturbation in the input can push the example into a region with very different loss contours. Because the feature importance is closely related to the gradient which is perpendicular to the loss contours, the importance scores can also be dramatically different. We provide additional analysis of this in Section 5. This first class of methods explains predictions in terms of the relative importance of features in a test input sample. Given the sample x t ∈ R d and the network's prediction l, we define the score of the predicted class S l (x t) to be the value of the l-th output neuron right before the softmax operation. We take l to be the class with the max score; i.e. the predicted class. Feature-importance methods seek to find the dimensions of input data point that most strongly affect the score, and in doing so, these methods assign an absolute saliency score to each input feature. Here we normalize the scores for each image by the sum of the saliency scores across the features. This ensures that any perturbations that we design change not the absolute feature saliencies (which may still preserve DISPLAYFORM0 This training point has a large influence on the loss at + This training point has a large influence on the loss at Figure 2 : Intuition for why interpretation is fragile. Consider a test example x t ∈ R 2 (black dot) that is slightly perturbed to a new position x t + δ in input space (gray dot). The contours and decision boundary corresponding to a loss function (L) for a two-class classification task are also shown, allowing one to see the direction of the gradient of the loss with respect to the input space. Neural networks with many parameters have decision boundaries that are roughly piecewise linear with many transitions. We illustrate that points near the transitions are especially fragile to interpretability-based analysis. A small perturbation to the input changes the direction of ∇ x L from being in the direction of x 1 to being in the direction of x 2, directly affecting feature-importance analyses. Similarly, a small perturbation to the test image changes which training image, when up-weighted, has the largest influence on L, directly affecting exemplar-based analysis.the ranking of different features), but their relative values. We summarize three different methods to calculate the normalized saliency score, denoted by R(x t).Simple gradient method Introduced in BID2 and applied to deep neural networks in , the simple gradient method applies a local linear approximation of the model to detect the sensitivity of the score to perturbing each of the input dimensions. Given input x t ∈ R d, the score is defined as: DISPLAYFORM0 Integrated gradients A significant drawback of the simple gradient method is the saturation problem discussed by;. introduced the integrated gradients method where the gradients of the score with respect to M scaled versions of the input are summed and then multiplied by the input. Letting x 0 be the reference point and ∆x t = x t − x 0, the feature importance vector is calculated by: DISPLAYFORM1 which is then normalized for our analysis. Here the absolute value is taken for each dimension. DeepLIFT DeepLIFT is an improved version of layer-wise relevance propagation (LRP) method BID1. LRP methods decompose the score S l (x t) backwards through the neural network. In each step, the score from the last layer is propagated to the previous layer, with the score being divided proportionally to magnitude of the activations of the neurons in the previous layer. The scores are propagated to the input layer, and the is a relevance score assigned to each of the input dimensions. DeepLIFT defines a reference point in the input space and propagates relevance scores proportionally to the changes in the neuronal activations from the reference. We use DeepLIFT with the Rescale rule; see for details. A complementary approach to interpreting the of a neural network is to explain the prediction of the network in terms of its training examples, {(x i, y i)}. DISPLAYFORM0 where z i def = (x i, y i) and z t is defined analogously. L(z,θ) is the loss of the network with parameters set toθ for the (training or test) data point z. Hθ DISPLAYFORM1 is the empirical Hessian of the network calculated over the training examples. The training examples with the highest influence are understood as explaining why a network made a particular prediction for a test example. We consider two natural metrics for quantifying the similarity between interpretations for two different images. As shown in Fig. 3, these metrics can be used to evaluate the effectiveness of a targeted attack on interpretability.• Spearman's rank order correlation: Because interpretation methods rank all of the features or training examples in order of importance, it is natural to use the rank correlation to compare the similarity between interpretations.• Top-k intersection: In many settings, only the most important features or interpretations are of interest. In these settings, we can compute the size of the intersection of the k most important features before and after perturbation. Problem statement For a given fixed neural network N and input data point x t, the feature importance and influence function methods that we have described produce an interpretation I(x t ; N). For feature importance, I(x t ; N) is a vector of feature scores; for influence function I(x t ; N) is a vector of scores for training examples. We would like to devise efficient perturbations to change the interpretability of a test image. Yet, the perturbations should be visually imperceptible and should not change the label of the prediction. Formally, we define the problem as: DISPLAYFORM0 where D(·) measures the change in interpretation (e.g. how many of the top-k pixels are no longer the top-k pixels of the saliency map after the perturbation) and > 0 constrains the norm of the perturbation. In this paper, we carry out three kinds of input perturbations. Random sign perturbation As a baseline, we generate random perturbations in which each pixel is randomly perturbed by ±. This is used to measure robustness against untargeted perturbations. Iterative attacks against feature-importance methods In Algorithm 1 we define two adversarial attacks against feature-importance methods, each of which consists of taking a series of steps in the direction that maximizes a differentiable dissimilarity function between the original and perturbed interpretation. The top-k attack seeks to perturb the saliency map by decreasing the relative importance of the k most important features of the original image. When the input data are images, the center of mass of the saliency map often captures the user's attention. The mass-center attack is designed to in the maximum spatial displacement of the center of mass of the saliency scores. Both of these attacks can be applied to any of the three feature-importance methods. We can obtain effective adversarial images for influence functions without resorting to interative procedures. We linearize FORMULA3 around the values of the current inputs and parameters. If we further constrain the L ∞ norm of the perturbation to, we obtain an optimal single-step perturbation: DISPLAYFORM0 Algorithm 1 Iterative Feature-Importance Attacks Input: test image x t, maximum norm of perturbation, normalized feature importance function R(·), number of iterations P, step size α Define a dissimilarity function D to measure the change between interpretations of two images: DISPLAYFORM1 where B is the set of the k largest dimensions a of R(x t), and C(·) is the center of saliency mass b. DISPLAYFORM2 Perturb the test image in the direction of signed gradient c of the dissimilarity function: DISPLAYFORM3 If needed, clip the perturbed input to satisfy the norm constraint: DISPLAYFORM4.., x P }, return the element with the largest value for the dissimilarity function and the same prediction as the original test image.a The goal is to damp the saliency scores of the k features originally identified as the most important. b The center of mass is defined for a W × H image as: DISPLAYFORM5 In some networks, such as those with ReLUs, this gradient is always 0. To attack interpretability in such networks, we replace the ReLU activations with their smooth approximation (softplus) when calculating the gradient and generate the perturbed image using this approximation. The perturbed images that are effective adversarial attacks against the original ReLU network, as discussed in Section 4.The attack we use consists of applying the negative of the perturbation in to decrease the influence of the 3 most influential training images of the original test image 1. Of course, this affects the influence of all of the other training images as well. We follow the same setup for computing the influence function as was done by the authors of. Because the influence is only calculated with respect to the parameters that change during training, we calculate the gradients only with respect to parameters in the final layer of our network (InceptionNet, see Section 4). This makes it feasible for us to compute exactly, but it gives us the perturbation of the input into the final layer, not the first layer. So, we use standard back-propagation to calculate the corresponding gradient for the input test image. We then take the sign of this gradient as the perturbation and clip the image to produce the adversarial test image. Data sets and models To evaluate the robustness of feature-importance methods, we used two image classification data sets: ILSVRC2012 (ImageNet classification challenge data set) and CIFAR-10 . For the ImageNet classification data set, we used a pre-trained SqueezeNet 2 model introduced by BID5. For the CIFAR-10 data set we trained our own convolutional network, whose architecture is presented in Appendix A.1 In other words, we generate the perturbation given by: − sign(DISPLAYFORM0 where z (i) is the i th most influential training image of the original test image. 2 https://github.com/rcmalli/keras-squeezenet For both data sets, the are examined using simple gradient, integrated gradients, and DeepLIFT feature importance methods. For DeepLIFT, we used the pixel-wise and the channelwise mean images as the CIFAR-10 and ImageNet reference points respectively. For the integrated gradients method, the same references were used with parameter M=100. We ran all iterative attack algorithms for P = 300 iterations with step size α = 0.5.To evaluate the robustness of influence functions, we followed a similar experimental setup to that of the original authors: we trained an InceptionNet v3 with all but the last layer frozen (the weights were pre-trained on ImageNet and obtained from Keras 3). The last layer was trained on a binary flower classification task (roses vs. sunflowers), using a data set consisting of 1,000 training images 4. This data set was chosen because it consisted of images that the network had not seen during pre-training on ImageNet. The network achieved a validation accuracy of 97.5% on this task. Results for feature-importance methods From the ImageNet test set, 512 correctly-classified images were randomly sampled for evaluation purposes. Examples of the mass-center attack against three feature importance methods were presented in FIG0 Figure 3: Evaluation metrics vs subjective change We generate snapshots of the perturbed image and its simple gradient saliency maps along with iterations of mass-center attack to visualize the gradual change in saliency map with its corresponding the rank-correlation and top-1000 intersection metrics. In FIG1, we present aggregated over all 512 images. We compare different attack methods using top-1000 intersection and rank correlation methods. In all the images, the attacks does not change the original predicted label of the image. Random sign perturbation already causes significant changes in both top-1000 intersection and rank order correlation. For example, with L ∞ = 8, on average, there is less than 30% overlap in the top 1000 most salient pixels between the original and the randomly perturbed images across all three of interpretation methods. This suggests that the saliency of individual or small groups of pixels can be extremely fragile to the input and should be interpreted with caution. With targeted perturbations, we observe more dramatic fragility. Even with a perturbation of L ∞ = 2, the interpretations change significantly. Both iterative attack algorithms have similar effects on feature importance of test images when measured on the basis of rank correlation or top-1000 intersection. In Appendix D, we show an additional metric: the displacement of the center of mass between the original and perturbed saliency maps. Empirically, we find this metric to correspond most strongly with intuitive perceptions of the similarity between two saliency maps. Not surprisingly, we found that the center attack method was more effective than the top-k attack at moving the center of mass of the saliency maps. Comparing the fragility of neural network interpretation among the three different methods, we found that the integrated gradients method was Across 512 correctly-classified ImageNet images, we find that the top-k and center attacks perform similarly in top-1000 intersection and rank correlation measures, and are far more effective than the random sign perturbation at demonstrating the fragility of interpretability, as characterized through top-1000 intersection (top) as well as rank order correlation (bottom). This is true for (a) the simple gradient method, (b) DeepLift, and (c) the integrated gradients method.the most robust to both random and adversarial attacks. Similar for CIFAR-10 can be found in Appendix D.Results for influence functions We evaluate the robustness of influence functions on a test data set consisting of 200 images of roses and sunflowers. Fig. 5 shows a representative test image to which we have applied the gradient sign attack. Although the prediction of the image does not change, the most influential training examples selected according to, as explanation for the prediction, change entirely from images of sunflowers and yellow petals that resemble the input image to those of red and pink roses that do not. Additional examples can be found in Appendix E.In Fig. 6, we compare the random perturbations and gradient sign attacks across all of the test images. We find that the gradient sign-based attacks are significantly more effective at decreasing the rank correlation of the influence of the training images, as well as distorting the top-5 influential images. For example, on average, with a targeted perturbation of magnitude = 8, only 2 of the top 5 most influential training images remain as the top 5 most influential images after the visually imperceptible perturbation. The influences of the training images before and after an adversarial attack are essentially uncorrelated. However, we find that even random attacks can have a nonnegligible effect on influence functions, on average reducing the rank correlation to 0.8 (≈ 10). In this section, we try to understand the source of interpretation fragility. The question is whether fragility a consequence of the complex non-linearities of a deep network or a characteristic present even in high-dimensional linear models, as is the case for adversarial examples for prediction BID4. To gain more insight into the fragility of gradient based interpretations, let S(x; W) denote the score function of interest; x ∈ R d is an input vector and W is the weights of the neural network, which is fixed since the network has finished training. We are interested in the Figure 5: Gradient sign attack on influence functions. An imperceptible perturbation to a test image can significantly affect exemplar-based interpretability. The original test image is that of a sunflower that is classified correctly in a rose vs. sunflower classification task. The top 3 training images identified by influence functions are shown in the top row. Using the gradient sign attack, we perturb the test image (with = 8) to produce the leftmost image in the second row. Although the image is even more confidently predicted as a sunflower, influence functions suggest very different training images by means of explanation: instead of the sunflowers and yellow petals that resemble the input image, the most influential images are pink/red roses. The plot on the right shows the influence of each training image before and after perturbation. The 3 most influential images (targeted by the attack) have decreased in influence, but the influences of other images have also changed. Figure 6: Comparison of random and targeted perturbations on influence functions. Here, we show the averaged of applying random (green) and gradient sign-based (orange) perturbations to 200 test images on the flower classification task. While random attacks affect interpretability, the effect is small and generally doesn't affect the most influential images. On the other hard, a targeted attack can significantly affect (a) the rank correlation and (b) even change the make-up of the 5 most influential images. Even at the maximal level of noise, the changes to the perturbed images were visually imperceptible, and prediction confidence was not significantly changed (the mean change was < 1% for random attacks and < 5% for targeted attacks at the highest level of noise).Hessian H whose entries are H i,j = ∂S ∂xi∂xj. The reason is that the first order approximation of gradient for some input perturbation direction δ ∈ R d is: DISPLAYFORM0 First, consider a linear model whose score for an input x is S = w x. Here, ∇ x S = w and ∇ 2 x S = 0; the feature-importance vector w is robust, because it is completely independent of x. Thus, some non-linearity is required for interpretation fragility. A simple network that is susceptible to adversarial attacks on interpretations consists of a set of weights connecting the input to a single neuron followed by a non-linearity (e.g. softmax): S = g(w x).We can calculate the change in saliency map due to a small perturbation in x → x + δ. The first-order approximation for the change in saliency map will be equal to: H · δ = ∇ 2 x S · δ. In particular, the saliency of the i th feature changes by (∇ 2 x S · δ) i and furthermore, the relative change DISPLAYFORM1 For the simple network, this relative change is: DISPLAYFORM2 where we have used g (·) and g (·) to refer to the first and second derivatives of g(·). Note that g (w x) and g (w x) do not scale with the dimensionality of x because in general, independent from the dimensionality, x and w are 2 -normalized or have fixed 2 -norm due to data preprocessing and weight decay regularization. However, if we choose δ = sign(w), then the relative change in the saliency grows with the dimension, since it is proportional to the 1 -norm of w. When the input is high-dimensional-which is the case with images-the relative effect of the perturbation can be substantial. Note also that this perturbation is exactly the sign of the first right singular vector of the Hessian ∇ 2 x S, which is appropriate since that is the vector that has the maximum effect on the gradient of S. A similar analysis can be carried out for influence functions (see Appendix F).For this simple network, the direction of adversarial attack on interpretability, sign(w) is the same as the adversarial attack on prediction. This means that we cannot perturb interpretability independently of prediction. For more complex networks, this is not the case and in Appendix G we show this analytically for a simple case of a two-layer network. As an empirical test, in FIG3, we plot the distribution of the angle between ∇ x S and v 1 (the first right singular vector of H which is the most fragile direction of feature importance) for 1000 CIFAR10 images (Details of the network in Appendix A). In FIG3, we plot the equivalent distribution for influence functions, computed across all 200 test images. The confirms that the steepest direction of change in interpretation and prediction are generally orthogonal, justifying how the perturbations can change the interpretation without changing the prediction. Related works To the best of our knowledge, the notion of adversarial examples has not previously been studied in the context of interpretation of neural networks. Adversarial attacks to the input that changes the prediction of a network have been actively studied. demonstrated that it is relatively easy to fool neural networks into making very different predictions for test images that are visually very similar to each other. BID4 introduced the Fast Gradient Sign Method (FGSM) as a one-step prediction attack. This was followed by more effective iterative attacks Interpretation of neural network predictions is also an active research area. Post-hoc interpretability is one family of methods that seek to "explain" the prediction without talking about the details of black-box model's hidden mechanisms. These included tools to explain predictions by networks in terms of the features of the test example (; ; ;), as well as in terms of contribution of training examples to the prediction at test time . These interpretations have gained increasing popularity, as they confer a degree of insight to human users of what the neural network might be doing .Conclusion This paper demonstrates that interpretation of neural networks can be fragile in the specific sense that two similar inputs with the same predicted label can be given very different interpretations. We develop new perturbations to illustrate this fragility and propose evaluation metrics as well as insights on why fragility occurs. Fragility of neural network interpretation is orthogonal to fragility of the prediction-we demonstrate how perturbations can substantially change the interpretation without changing the predicted label. The two types of fragility do arise from similar factors, as we discuss in Section 5. Our focus is on the interpretation method, rather than on the original network, and as such we do not explore how interpretable is the original predictor. There is a separately line of research that tries to design simpler and more interpretable prediction models BID0.Our main message is that robustness of the interpretation of a prediction is an important and challenging problem, especially as in many applications (e.g. many biomedical and social settings) users are as interested in the interpretation as in the prediction itself. Our raise concerns on how interpretations of neural networks are sensitive to noise and can be manipulated. Especially in settings where the importance of individual or a small subset of features are interpreted, we show that these importance scores can be sensitive to even random perturbation. More dramatic manipulations of interpretations can be achieved with our targeted perturbations, which raise security concerns. We do not suggest that interpretations are meaningless, just as adversarial attacks on predictions do not imply that neural networks are useless. Interpretation methods do need to be used and evaluated with caution while applied to neural networks, as they can be fooled into identifying features that would not be considered salient by human perception. Our demonstrate that the interpretations (e.g. saliency maps) are vulnerable to perturbations, but this does not imply that the interpretation methods are broken by the perturbations. This is a subtle but important distinction. Methods such as saliency measure the infinitesimal sensitivity of the neural network at a particular input x. After a perturbation, the input has changed tox = x + δ, and the salency now measures the sensitivity at the perturbed input. The saliency correctly captures the infinitesimal sensitivity at the two inputs; it's doing what it is supposed to do. The fact that the two ing saliency maps are very different is fundamentally due to the network itself being fragile to such perturbations, as we illustrate with Fig. 2.While we focus on image data (ImageNet and CIFAR-10), because these are the standard benchmarks for popular interpretation tools, this fragility issue can be wide-spread in biomedical, economic and other settings where neural networks are increasingly used. Understanding interpretation fragility in these applications and develop more robust methods are important agendas of research. We trained the following structure using ADAM optimizer with default parameters. The ing test accuracy using ReLU activation was 73%. For the experiment in FIG3, we replaced ReLU activation with Softplus and retrained the network (with the ReLU network weights as initial weights). The ing accuracy was 73%. 3 × 3 conv. 96 ReLU 3 × 3 conv. 96 ReLU 3 × 3 conv. 96 Relu Stride 2 3 × 3 conv. 192 ReLU 3 × 3 conv. 192 ReLU 3 × 3 conv. 192 Relu Stride 2 1024 hidden sized feed forward Here we provide three more examples from ImageNet. For each example, all three methods of feature importance are attacked by random sign noise and our two targeted adversarial algorithms. Figure 12: Center-shift for three feature importance methods on ImageNet: As discussed in the paper, among our three measurements, center-shift measure was the most correlated measure with the subjective perception of change in saliency maps. The in Appendix B also show that the center attack which ed in largest average center-shift, also in the most significant subjective change in saliency maps. Random sign perturbations, on the other side, did not substantially change the global shape of the saliency maps, though local pockets of saliency are sensitive. Just like rank correlation and top-1000 intersection measures, the integrated gradients method is the most robust method against adversarial attacks in the center-shift measure. FIG0: Results for adversarial attacks against CIFAR10 feature importance methods: For CIFAR10 the mass-center attack and top-k attack with k=100 achieve similar for rank correlation and top-100 intersection measurements and both are stronger than random perturbations. Mass-center attack moves the center of mass more than two other perturbations. Among different feature importance methods, integrated gradients is more robust than the two other methods. Additionally, for CIFAR10 show that images in this data set are more robust against adversarial attack compared to ImageNet images which agrees with our analysis that higher dimensional inputs are tend to be more fragile. In this appendix, we provide additional examples of the fragility of influence functions, analogous to Fig. 5. Here, we demonstrate that increasing the dimension of the input of a simple neural network increases the fragility of that network with respect to influence functions, analogous to the calculations carried out for importance-feature methods in Section 5. Recall that the influence of a training image z i = (x i, y i) on a test image z = (x, y) is given by: DISPLAYFORM0 We restrict our attention to the term in that is dependent on x, and denote it by J def = ∇ θ L. J represents the infinitesimal effect of each of the parameters in the network on the loss function evaluated at the test image. Now, let us calculate the change in this term due to a small perturbation in x → x + δ. The firstorder approximation for the change in J is equal to: DISPLAYFORM1 For the simple network defined in Section 5, this evaluates to (replacing θ with w for consistency of notation): DISPLAYFORM2 where for simplicity, we have taken the loss to be L = |y − g(w x)|, making the derivatives easier to calculate. Furthermore, we have used g (·) and g (·) to refer to the first and second derivatives of g(·). Note that g (w x) and g (w x) do not scale with the dimensionality of x because x and w are generalized L 2 -normalized due to data preprocessing and weight decay regularization. However, if we choose δ = sign(w), then the relative change in the saliency grows with the dimension, since it is proportional to the L 1 -norm of w. Consider a two layer neural network with activation function g(·), input x ∈ R d, hidden vector u ∈ R h, and score function S), we have: DISPLAYFORM0 where w j = ||w j || 2ŵj. We have: DISPLAYFORM1 Now for an input sample x perturbation δ, for the change in feature importance: DISPLAYFORM2 which is equal to: DISPLAYFORM3 We further assume that the input is high-dimensional so that h < d and for i = j we have w j · w i = 0. For maximizing the 2 norm of saliency difference we have the following perturbation direction:δ m = argmax ||δ||=1 ||∇ x S(x + δ) − ∇ x S(x)|| =ŵ k where: k = argmax|v j g (w j .x)| × ||w k || 2 2 comparing which to the direction of feature importance: DISPLAYFORM4 we conclude that the two directions are not parallel unless g = g which is not the case for many activation functions like Softplus, Sigmoid, etc. The analyses and experiments in this paper have demonstrated that small perturbations in the input layers of deep neural networks can have large changes in the interpretations. This is analogous to classical adversarial examples, whereby small perturbations in the input produce large changes in the prediction. In that setting, it has been proposed that the Lipschitz constant of the network be constrained during training to limit the effect of adversarial perturbations . This has found some empirical success BID3.Here, we propose an analogous method to upper-bound the change in interpretability of a neural network as a of perturbations to the input. Specifically, consider a network with K layers, which takes as input a data point we denote as y 0. The output of the i th layer is given by y i+1 = f i (y i) for i = 0, 1... K − 1. We define S def = f K−1 (f K−2 (. . . f 0 (y 0)...)) to be the output (e.g. score for the correct class) of our network, and we are interested in designing a network whose gradient S = ∇ y0 S is relatively insensitive to perturbations in the input, as this corresponds to a network whose feature importances are robust. A natural quantity to consider is the Lipschitz constant of S with respect to y 0. By the chain rule, the Lipschitz constant of S is DISPLAYFORM0 Now consider the function f i (·), which maps y i to y i+1. In the simple case of the fully-connected network, which we consider here, f i (y i) = g i (W i y i), where g i is a non-linearity and W i are the trained weights for that layer. Thus, the Lipschitz constant of the i th partial derivative in FORMULA24 is the Lipschitz constant of DISPLAYFORM1 which is upper-bounded by ||W i || 2 · L(g i (·)), where ||W || denotes the operator norm of W (its largest singular value) 5. This suggests that a conservative upper ceiling for is DISPLAYFORM2 Because the Lipschitz constant of the non-linearities g i (·) are fixed, this suggests that a regularization based on the operator norms of the weights W i may allow us to train networks that are robust to attacks on feature importance. The calculations in this Appendix section is meant to be suggestive rather than conclusive, since in practice the Lipschitz bounds are rarely tight.

Can we trust a neural network's explanation for its prediction? We examine the robustness of several popular notions of interpretability of neural networks including saliency maps and influence functions and design adversarial examples against them.

Represent the TLDR for finding the corresponding scientific article

We introduce a type of neural network that is structurally resistant to adversarial attacks, even when trained on unaugmented training sets.  The resistance is due to the stability of network units wrt input perturbations.