Question: Represent the text to find a very short summary (1-2 sentences max)

Question: A collection of scientific papers is often accompanied by tags: keywords, topics, concepts etc., associated with each paper. Sometimes these tags are human-generated, sometimes they are machine-generated. We propose a simple measure of the consistency of the tagging of scientific papers: whether these tags are predictive for the citation graph links. Since the authors tend to cite papers about the topics close to those of their publications, a consistent tagging system could predict citations. We present an algorithm to calculate consistency, and experiments with human- and machine-generated tags. We show that augmentation, i.e. the combination of the manual tags with the machine-generated ones, can enhance the consistency of the tags. We further introduce cross-consistency, the ability to predict citation links between papers tagged by different taggers, e.g. manually and by a machine. Cross-consistency can be used to evaluate the tagging quality when the amount of labeled data is limited. A part of a construction of a knowledge graph is the analysis of publications and adding to them tags: concept names, keywords, etc. This often involves natural language processing or other machine learning methods BID8. To develop such methods one must have a measure of success: one should be able to determine whether the given tagging is "good" or "bad". The most direct way to test the machine produced tags is to compare them to the tags produced by humans. One creates a "golden set" of papers tagged by humans, and penalizes the algorithms for any deviation from these tags. There are, however, certain problems with this approach. First, human tagging is expensiveeven more so for scientific papers, where human taggers must have a specialized training just to understand what the papers are about. Second, even the best human taggers' are inconsistent. This provides a natural limitation for this method BID7. The latter problem is exacerbated when the tagging dictionary is large. For example, the popular US National Library of Medicine database of Medical Subject Headings (MeSH, https://www.nlm.nih.gov/mesh/) has just under 30 000 entries. A superset of MeSH, Unified Medical Language System (UMLS, https://www.nlm.nih.gov/research/ umls/knowledge_sources/metathesaurus/release/statistics.html) contains a staggering amount of 3 822 832 distinct concepts. It is doubtful a human can do a good job choosing the right tags from a dictionary so large. A domain expert usually deals with a subsystem of the dictionary, covering her area of expertise. This presents obvious difficulties for tagging papers about multidisciplinary research, that may require a combination of the efforts of several highly qualified taggers. Another problem is the evaluation of tag augmentation. Suppose we have papers tagged by humans, and we want to add machine-generated tags, for example, to improve the search function in the collection. Do the new tags actually add to the quality or subtract from it? How can we evaluate the if our tags are by definition different from those produced by humans?Thus a measure of the tagging quality other than a direct comparison with manually produced tags may be useful for the assessing the work of the tagging engines. This is especially important for an ongoing quality control of an engine that continuously ingests and tags fresh publications. In this paper we propose such a measure. The idea for this measure is inspired by the works on graph embeddings [, Grover and BID3 . In these works one tags graph nodes and compares different sets of tags. The usual comparison criterion is whether the tags can predict graph edges: nodes connected by an edge should have similar tags, while nodes not connected by an edge should have dissimilar tags. To use this approach we need to represent papers as nodes on a graph. A natural choice is the citation graph: and edge from paper A to paper B means that paper A cites paper B. This leads to the following assumptions:1. Scientific papers cited by the given paper A are more similar to A than the other (non cited) papers.2. A good tagging system must reflect this. In other words, a good set of tags must be able to predict links on the citation graph, and the quality of the prediction reflects the quality of the tags. We will call this property consistency: a good tagger consistently gives similar tags to similar papers. It is worth stressing that consistency is just one component of the quality of a tagger. If a tagger consistently uses keyword library instead of keyword bread BID0, this measure would give it high marks, despite tags being obviously wrong. A way to overcome this deficiency is to calculate cross-consistency with a known "good" tagger. For example, we can tag some papers manually, and some papers using machine generated tags, and then predict citation links between these papers. This cross-consistency measures the similarity between these taggers. This application is interesting because it allows us to expand the number of labeled papers for evaluation of machine-based taggers. We can create a golden set of manually tagged papers, and then generate tags for the papers in their reference lists, and the random samples using the machine-based tagger. Since a typical paper cites many publications, this approach significantly expands the quantity of data available for training and testing. To create a measure based on these ideas one should note that citation links strongly depend on the time the candidate for citation was published. Even a very relevant paper may not be cited if it is too old or too new. In the first case the citing authors may prefer a newer paper on the same topic. In the second case they may overlook the most recent publications. Therefore we recast our assumptions in the following way:A consistent tagging system should be able to predict citation links from a given paper to a set of simultaneously published papers. The rest of the paper is organized as follows. In Section 2 we discuss the algorithm to calculate the consistency of the given tagging system. Experiments with this measure are discussed in Section 3. In Section 4 we present the . The algorithm to calculate the consistency measure is shown as Algorithm 1. We select n seed papers. For each seed paper we take up to k random papers from its reference list, and label them with the label y = 1. These are our cited papers, or positive samples. For each cited paper we randomly choose k papers from the corpus with the same publication date as the cited paper. These are our negative samples FIG0 ). The notion of "publication date" depends on the granularity g of publication dates. We label the chosen negative samples with the label y = 0. Now we have a vector of labels y i of size k + km, with k ones and km zeros. We calculate tags for the seed paper, positive and negative samples, and the number of overlapping tags between the seed paper and each of positive and negative samples. This gives us k + km overlap numbers t i of each seed paper. We can solve the one dimension classification problem y ∼ t and calculate its ROC curve BID2. The area under curve (AUC) for this problem is the measure of the consistency of the tagging. The average AUC and its variation are the numbers we are interested in. Algorithm 1 can be used for calculation of consistency or cross-consistency of taggers. In the latter case we just choose different sources of tags for seed papers and samples. For experiments we used the papers extracted from the PubMed database (https://www. ncbi.nlm.nih.gov/pubmed/). These papers have MeSH tags attached manually by the human taggers. We added to them additional tags by processing papers' titles and abstracts using several packages. Gene names were identified using GNAT BID4, diseases were identified using DNORM BID6, and additional UMLS concepts Algorithm 1 Calculation of consistency measure for the given tagging system Input: Parameters: n seed papers, k cited papers per seed paper, m negative samples per cited paper, granularity g of the publication date 1: for all seed papers s do Select min(k, bibliography length) random papers from the reference list of s (cited papers or positive samples). Label them as y i = 1 for all cited papers c do Select m random papers with same publication date as c, using date granularity g (negative samples). Label them as y i = 0. Tag the seed paper, positive and negative samples. for all positive and negative samples p do Calculate the number t i of overlapping tags between the seed paper s and the sample p 9: end for Calculate AUC for the classification problem y ∼ t 11: end for 12: return The set of AUCs. The average AUC is the consistency measure, while the variation provides the error estimate were mapped using NEJI BID1. The coverage of papers by different tagging sources is shown in Table 1.The number of seed papers n in Algorithm 1 was chosen to be n = 100. Based on the preliminary experiments we chose the following hyperparameters which produced a good convergence of the measure: the number of cited papers per see paper k = 10, the number of negative samples per cited paper m = 2. The date granularity chosen was year-month: two papers were considered published at the same time if their years and months of publication in PubMed coincided. The baseline to compare the against was constructed by randomizing the tag sets. We linked each paper (seeds, references, and negative samples) to the tags from all the sources, and then randomly shuffled the papers, so each paper got a set of tags from some other paper in our sample. We expect the average AUC for this random tag set to be 0.5, reflecting the lack of discrimination between the cited papers and negative samples. We calculate the AUC for each of our seed papers, and report the distribution of the in Tukey "box and whiskers" plots. On Figure 2 we show the consistency measure for the tagging sources: DNORM, GNAT, MANUAL, NEJI as well as the combined automatic taggers (NEJI + DNORM + GNAT).Adding machine-generated tags to the manual ones is explored on Figure 3. Here we take manually created tags and add machine-generated ones from different sources, again using random tags as a baseline. On Figure 4 we show cross-consistency between the manual tags and NEJI-generated ones (since GNAT and DNORM are used to predict only specific concepts like genes and diseases, they are omitted from the experiment). We used one source for tagging seed papers, and another source for tagging samples (cited papers and negative samples). First, there is clear difference between the consistency of the randomly generated tags and the real ones (Figure 2). As expected, the consistency of the random tags is concentrated at AUC = 0.5, with some outliers both above and below this value. In contrast, the consistency of the real tags is almost always above AUC = 0.5. An exception is tagging sources of low coverage like GNAT (see Table 1), where consistency is close to 0.5. Obviously when the coverage is low, most positive and negative samples have zero overlap with their seed papers, which lowers AUC. Unexpectedly, the consistency of high coverage machine generated sources like NEJI is on par with the human tags. Tags augmentation is explored on Figure 3. As expected, adding random tags to the manually generated ones does not noticeably change the consistency of the . However, adding "real" machine generated tags is improving our measure, which is another evidence that the measure itself is reasonable. The cross-consistency between manual tags and machine-generated ones is shown on Figure 4. Here we used different sources for seed papers and for samples. While crossconsistency is lower than the internal consistency of each tagger, is still is significantly higher than for random tags. In , a simple measure of consistency of tagging: whether it is predictive for citation links in a knowledge graph,-seems to be informative about the tagging process and can be used, along with other measures, to assess and evaluate it. Cross-consistency between different taggers can be used to estimate their similarity, especially when some taggers (e.g. manual tagging) are too expensive to run on a large set of papers. Cross consistency between manual tags and NEJI generated ones. X axis shows the source for the seed papers, Y axes shows the source for samples

A good tagger gives similar tags to a given paper and the papers it cites

represent the short summary for finding the corresponding scientific article (long).
To give you a sense - "We propose an approach to learn decentralized policies in multi-agent settings using attention-based critics and demonstrate promising results in environments with complex interactions." should be close to "Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems Reinforcement learning has recently made exciting progress in many domains, including Atari games, the ancient Chinese board game, Go, and complex continuous control tasks involving locomotion BID17 BID26 BID12. While most reinforcement learning paradigms focus on single agents acting in a static environment (or against themselves in the case of Go), real-world agents often compete or cooperate with other agents in a dynamically shifting environment. In order to learn effectively in multi-agent environments, agents must not only learn the dynamics of their environment, but also those of the other learning agents present. To this end, several approaches for multi-agent reinforcement learning have been developed. The simplest approach is to train each agent independently to maximize their individual reward, while treating other agents as part of the environment. However, this approach violates the basic assumption underlying reinforcement learning, that the environment should be stationary and Markovian. Any single agent's environment is dynamic and nonstationary due to other agents' changing policies. As such, standard algorithms developed for stationary Markov decision processes fail. At the other end of the spectrum, all agents can be collectively modeled as a single-agent whose action space is the joint action space of all agents BID2. While allowing coordinated behaviors across agents, this approach is not scalable due to the action space size increasing exponentially with the number of agents. It also demands a high degree of communication during execution, as the central policy must collect observations from and distribute actions to the individual agents. In real-world settings, this demand can be problematic. Recent work BID20 attempts to combine the strengths of these two approaches. In particular, a critic (or a number of critics) is centrally learned with information from all agents. The actors, however, receive information only from their corresponding agents. Thus, during testing, executing the policies does not require the knowledge of other agents' actions. This paradigm circumvents the challenge of non-Markovian and non-stationary environments during learning. Despite those progresses, however, algorithms for multi-agent reinforcement learning are still far from being scalable (to a larger number of agents) and being generically applicable to environments and tasks that are co-operative (sharing a global reward), competitive, or mixed. Our approach extends these prior works in several directions. The main idea is to centrally learn a critic with an attention mechanism. The intuition behind our idea is that in many real-world environ-ments, it is beneficial for agents to know what other agents it should pay attention to. For example, a soccer defender needs to pay attention to attackers in their vicinity as well as the player with the ball, while she/he rarely needs to pay attention to the opposing team's goalie. The specific attackers that the defender is paying attention to can change at different parts of the game, depending on the formation and strategy of the opponent. A typical centralized approach to multi-agent reinforcement learning does not take these dynamics into account, instead simply considering all agents at all timepoints. Our attention mechanism is able to dynamically select which agents to attend to at each time point, improving performance in multi-agent domains with complex interactions. The proposed approach has an input space linearly increasing with respect to the number of agents, as opposed to the quadratic increase in a previous approach BID20. It also works well in co-operative, competitive, and mixed environments, exceeding the capability of some prior work that focuses only on co-operative environments.We have validated our approach on two simulated environments and tasks. We plan to release the code for both the model and the environments after the reviewing period ends. The rest of the paper is organized as follows. In section 2, we discuss related work, followed by a detailed description of our approach in section 3. We report experimental studies in section 4 and conclude in section 5. Multi-Agent Reinforcement Learning (MARL) is a long studied problem BID2. Topics within MARL are diverse, ranging from learning communication between cooperative agents (; BID4 to algorithms for optimal play in competitive settings BID19, though, until recently, they have been focused on simple gridworld environments with tabular learning methods. As deep learning based approaches to reinforcement learning have grown more popular, they have, naturally, been applied to the MARL setting BID32 BID9, allowing multi-agent learning in high-dimensional/continuous state spaces; however, naive applications of Deep RL methods to MARL naturally encounter some limitations, such as nonstationarity of the environment from the perspective of individual agents BID6 BID20, lack of coordination/communication in cooperative settings BID29 BID23 BID20 BID5, credit assignment in cooperative settings with global rewards BID25 BID30, and the failure to take opponent strategies into account when learning agent policies BID11 .Most relevant to this work are recent, non-attention approaches that propose an actor-critic framework consisting of centralized training with decentralized execution BID20, as well as some approaches that utilize attention in a fully centralized multi-agent setting BID3 BID14 . BID20 investigate the challenges of multiagent learning in mixed reward environments BID2 . They propose an actor-critic method that uses separate centralized critics for each agent which take in all other agents' actions and observations as input, while training policies that are conditioned only on local information. This practice reduces the non-stationarity of multi-agent environments, as considering the actions of other agents to be part of the environment makes the state transition dynamics stable from the perspective of one agent. In practice, these ideas greatly stabilize learning, due to reduced variance in the value function estimates. Similarly introduce a centralized critic for cooperative settings with shared rewards. Their method incorporates a "counterfactual baseline" for calculating the advantage function which is able to marginalize a single agent's actions while keeping others fixed. This method allows for complex multi-agent credit assignment, as the advantage function only encourages actions that directly influence an agent's rewards. Attention models have recently emerged as a successful approach to intelligently selecting contextual information, with applications in computer vision BID0 BID21, natural language processing (; BID1 BID18, and reinforcement learning BID24 .In a similar vein, BID14 proposed an attention-based actor-critic algorithm for MARL. This work follows the alternative paradigm of centralizing policies while keeping the critics decentralized. Their focus is on learning an attention model for sharing information between the policies. As such, this approach is complementary to ours, and a combination of both approaches could yield further performance benefits in cases where centralized policies are desirable. Our proposed approach is more flexible than the aformentioned approaches for MARL. Our algorithm is able to train policies in environments with any reward setup, different action spaces for each agent, a variance-reducing baseline that only marginalizes the relevant agent's actions, and with a set of centralized critics that dynamically attend to the relevant information for each agent at each time point. As such, our approach is more scalable to the number of agents, and is more broadly applicable to different types of environments. We start by introducing the necessary notation and basic building blocks for our approach. We then describe our ideas in detail. We consider the framework of Markov Games BID19, which is a multi-agent extension of Markov Decision Processes. They are defined by a set of states, S, action sets for each of N agents, A 1, ..., A N, a state transition function, T : S × A 1 × ... × A N → P (S), which defines the probability distribution over possible next states, given the current state and actions for each agent, and a reward function for each agent that also depends on the global state and actions of all agents, DISPLAYFORM0 We will specifically be considering a partially observable variant in which an agent, i receives an observation, o i, which contains partial information from the global state, s ∈ S. Each agent learns a policy, π i: O i → P (A i) which maps each agent's observation to a distribution over it's set of actions. The agents aim to learn a policy that maximizes their expected discounted returns, DISPLAYFORM1, where γ ∈ is the discount factor that determines how much the policy favors immediate reward over long-term gain. Policy Gradients Policy gradient techniques BID31 ) aim to estimate the gradient of an agent's expected returns with respect to the parameters of its policy. This gradient estimate takes the following form: DISPLAYFORM2 Actor-Critic and Soft Actor-Critic The term ∞ t =t γ t −t r t (s t, a t) in the policy gradient estimator leads to high variance, as these returns can vary drastically between episodes. Actor-critic methods BID16 aim to ameliorate this issue by using a function approximation of the expected returns, and replacing the original return term in the policy gradient estimator with this function. One specific instance of actor-critic methods learns a function to estimate expected discounted returns, given a state and action, Q ψ (s t, a t) = E[DISPLAYFORM3, learned through temporal-difference learning by minimizing the regression loss: DISPLAYFORM4 where Qψ is the target Q-value function. To encourage exploration and avoid converging to non-optimal deterministic policies, recent approaches of maximum entropy reinforcement learning learn a soft value function by modifying the policy gradient to incorporate an entropy term BID10 : DISPLAYFORM5 where b(s) is a state-dependent baseline (for the Q-value function). The loss function for temporaldifference learning of the value function is also revised accordingly with a new target: DISPLAYFORM6 While an estimate of the value function V φ (s) can be used a baseline, we provide an alternative that further reduces variance and addresses credit assignment in the multi-agent setting in section 3.2. The main idea behind our multi-agent learning approach is to learn the critic for each agent by selectively paying attention to other agents' actions. This is the same paradigm of training critics centrally (to overcome the challenge of non-stationary non-Markovian environments) and executing learned policies distributedly. FIG0 illustrates the main components of our approach. Attention The attention mechanism functions in a manner similar to a differentiable key-value memory model BID24. Intuitively, each agent queries the other agents for information about their observations and actions and incorporates that information into the estimate of its value function. This paradigm was chosen, in contrast to other attention-based approaches, as it doesn't make any assumptions about the temporal or spatial locality of the inputs, as opposed to approaches taken in the natural language processing and computer vision fields. DISPLAYFORM0 where f i is a two-layer multi-layer perceptron (MLP), while g i is a one-layer MLP embedding function. The contribution from other agents, x i, is a weighted sum of each agent's value: DISPLAYFORM1 where the value, v j is a function of agent j's embedding, encoded with an embedding function and then linearly transformed by a shared matrix V. h is an element-wise nonlinearity (we have used leaky ReLU).The attention weight α j compares the embedding e j with e i = g i (o i, a i), using a bilinear mapping (ie, the query-key system) and passes the similarity value between these two embeddings into a softmax DISPLAYFORM2 where W q transforms e i into a "query" and W k transforms e j into a "key". The matching is then scaled by the dimensionality of these two matrices to prevent vanishing gradients .In our experiments, we have used multiple attention heads . In this case, each head, using a separate set of parameters (W k, W q, V), gives rise to an aggregated contribution from all other agents to the agent i and we simply concatenate the contributions from all heads as a single vector. Crucially, each head can focus on a different weighted mixture of agents. Note that the weights for extracting selectors, keys, and values are shared across all agents, which encourages a common embedding space. The sharing of critic parameters between agents is possible, even in adversarial settings, because multi-agent value-function approximation is, essentially, a multi-task regression problem. This method can easily be extended to include additional information, beyond local observations and actions, at training time, including the global state if it is available, simply by adding additional encoders, e. (We do not consider this case in our experiments, however, as our approach is effective in combining local observations to predict expected returns in environments where the global state may not be available).Learning with Attentive Critics All critics are updated together to minimize a joint regression loss function, due to the parameter sharing: DISPLAYFORM3, where DISPLAYFORM4 whereψ andθ are the parameters of the target critics and target policies respectively. Note that Q ψ i, the action-value estimate for agent i, receives observations and actions for all agents. α is the temperature parameter determining the balance between maximizing entropy and rewards. The individual policies are updated with the following gradient: DISPLAYFORM5 where b(o, a \i) is the multi-agent baseline used to calculate the advantage function decribed in the following section. Note that we are sampling all actions, a, from all agents' current policies in order to calculate the gradient estimate for agent i, unlike in the MADDPG algorithm BID20, where the other agents' actions are sampled from the replay buffer, potentially causing overgeneralization where agents fail to coordinate based on their current policies. Full training details and hyperparameters can be found in the appendix 6.1. As shown in, an advantage function using a baseline that only marginalizes out the actions of the given agent from Q ψ i (o, a), can help solve the multi-agent credit assignment problem. In other words, by comparing the value of a specific action to the value of the average action for the agent, with all other agents fixed, we can learn whether said action will cause an increase in expected return or whether any increase in reward is attributed to the actions of other agents. The form of this advantage function is shown below: DISPLAYFORM0 Using our attention mechanism, we can implement a more general and flexible form of a multiagent baseline that, unlike the advantage function proposed in, doesn't assume the same action space for each agent, doesn't require a global reward, and attends dynamically to other agents, as in our Q-function. This is made simple by the natural decomposition of an agents encoding, e i, and the weighted sum of encodings of other agents, x i, in our attention model. Concretely, in the case of discrete policies, we can calculate our baseline in a single forward pass by outputting the expected return Q i (o, (a i, a \i)) for every possible action, a i ∈ A i, that agent i can take. We can then calculate the expectation exactly: DISPLAYFORM1 In order to do so, we must remove a i from the input of Q i, and output a value for every action. We add an observation-encoder, e i = g o i (o i), for each agent, using these encodings in place of the e i = g i (o i, a i) described above, and modify f i such that it outputs a value for each possible action, rather than the single input action. In the case of continuous policies, we do not need to add any parameters, as we can simply estimate the expectation in Equation 9 by sampling actions from our policy and averaging their Q-values, though, this comes at the cost of multiple expensive passes through the network. 4.1 SETUP (a) Cooperative Treasure Collection. The small grey agents are "hunters" who collect the colored treasure, and deposit them with the correctly colored large "bank" agents.(b) Rover-Tower. Each grey "Tower" is paired with a "Rover" and a destination (color of rover corresponds to its destination). Their goal is to communicate with the "Rover" such that it moves toward the destination. We construct two environments that test various capabilities of our approach (MAAC) and baselines. We investigate in two main directions. First, we study the scalability of different methods as the number of agents grows. We hypothesize that the current approach of concatenating all agents' observations (often used as a global state to be shared among agents) and actions in order to centralize critics does not scale well. To this end, we implement a cooperative environment, Cooperative Treasure Collection, with shared rewards where we can vary the total number of agents. The experimental in sec 4.3 validate our claim. Secondly, we want to evaluate each method's ability to attend to information relevant to rewards. Moreover, the relevance (to rewards) can dynamically change during an episode. This is analogous to real-life tasks such as the soccer example presented earlier. To this end, we implement a Rover-Tower task environment where randomly paired agents communicate information and coordinate. The two environments are implemented in the multi-agent particle environment framework 1 introduced by BID23, and extended by BID20. We found this framework useful for creating environments involving complex interaction between agents, while keeping the control and perception problems simple, as we are primarily interested in addressing agent interaction. To further simplify the control problem, we use discrete action spaces, allowing agents to move up, down, left, right, or stay; however, the agents may not immediately move exactly in the specified direction, as the task framework incorporates a basic physics engine where agents' momentums are taken into account. Fig. 2 illustrates the two environments. Cooperative Treasure Collection The cooperative environment in Figure 2a ) involves 8 total agents, 6 of which are "treasure hunters" and 2 of which are "treasure banks", which each correspond to a different color of treasure. The role of the hunters is to collect the treasure of any color, which re-spawn randomly upon being collected (with a total of 6), and then "deposit" the treasure into the correctly colored "bank". The role of each bank is to simply gather as much treasure as possible from the hunters. All agents are able to see each others' positions with respect to their own. Hunters receive a global reward for the successful collection of treasure and all agents receive a global reward for the depositing of treasure. Hunters are additionally penalized for colliding with each other. As such, the task contains a mixture of shared and individual rewards and requires different "modes of attention" which depend on the agent's state and other agents' potential for affecting its rewards. Rover-Tower The environment in Figure 2b involves 8 total agents, 4 of which are "rovers" and another 4 which are "towers". At each episode, rovers and towers are randomly paired. The pair is negatively rewarded by the distance of the rover to its goal. The task can be thought of as a navigation task on an alien planet with limited infrastructure and low visibility. The rovers are unable to see in their surroundings and must rely on communication from the towers, which are able to locate the rovers as well as their destinations and can send one of five discrete communication messages to their paired rover. Note that communication is highly restricted and different from centralized policy approaches BID14, which allow for free transfer of continuous information among policies. In our setup, the communication is integrated into the environment (in the tower's action BID17 space and the rover's observation space), rather than being explicitly part of the model, and is limited to a few discrete signals. We compare to two recently proposed approaches for centralized training of decentralized policies: MADDPG BID20 and COMA, as well as a single-agent RL approach, DDPG, trained separately for each agent. In order to enable learning in discrete action spaces for both MADDPG and DDPG, where deterministic policies are not possible, we use the Gumbel-Softmax reparametrization trick BID13. We will refer to these modified versions as MADDPG (Discrete) and DDPG (Discrete). For a detailed description of this reparametrization, see the appendix 6.2. We use soft actor critic to optimize. Thus, in order to have fair comparisons, we additionally implement MADDPG and COMA with Soft Actor-Critic, named as MADDPG+SAC and COMA+SAC.We also consider an ablated version of our model as a variant of our approach. In this model, we use uniform attention by fixing the attention weight α j (Eq. 6) to be 1/(N − 1). This restriction prevents the model from focusing its attention on specific agents. All methods are implemented such that their approximate total number of parameters (across agents) are equal to our method, and each model is trained with 6 random seeds each. Hyperparameters for each underlying algorithm are tuned based on performance and kept constant across all variants of critic architectures for that algorithm. A thorough comparison of all baselines is summarized in TAB0. FIG1 illustrates averaged rewards per episode by various methods. The proposed approach (MAAC) is competitive with other approaches being compared. In what follows, we provide detailed analysis. Uniform attention is competitive with our approach in the Cooperative Treasure Collection (CTC) environment, but not in Rover-Tower. On the other hand, both MADDPG (Discrete) and MADDPG+SAC perform well on Rover-Tower, though they do not on CTC. Both variants of COMA do not fare well in our environments. DDPG, arguably a weaker baseline, performs surprisingly well in CTC, but does poorly in Rover-Tower. In CTC, the rewards are shared across agents thus an agent's critic does not need to focus on information from specific agents in order to calculate its expected rewards. Moreover, each agent's local observation provides enough information to make a decent prediction of its expected rewards. This might explain why MAAC (uniform) which attends to other agents equally, and DDPG (being very unattentive to other agents) perform well. On the other hand, rewards in the Rover-Tower environment for a specific agent are tied to another single agent's observations. This environment exemplifies a class of scenarios where dynamic attention can be beneficial: when subgroups of agents are interacting and performing coordinated tasks with separate rewards, but the groups do not remain static. This explains why MAAC (uniform) perform poorly and DDPG completely breaks down, as knowing information from another specific agent is crucial in predicting expected rewards. COMA uses a single centralized network for predicting Q-values for all agents with separate forward passes. Thus, this approach may perform best in environments with global rewards and agents with similar action spaces. However, our environments have agents with differing roles (and non-global rewards in the case of Rover-Tower). Thus both variants of COMA do not fare well. MADDPG (and its variant) is a very strong method. However, we suspect its low performance in CTC is due to this environment's relatively large observation spaces for all agents, as the MADDPG critic concatenates observations for all agents into a single input vector for each agent's critic. Our next experiments confirm this hypothesis. Scalability We compare the average rewards attained by both approaches (normalized by the range of rewards attained in the environment, as differing the number of agents changes the nature of rewards in each environment), and show that the improvement of our approach MAAC over MADDPG+SAC grows with respect to the number of agents. As suspected, MADDPG-like critics use all information non-selectively, while our approach can learn which agents to pay more attention through the attention mechanism. Thus our approach scales better when the number of agents increases. In future research we will continue to improve the scalability when the number of agents further increases by sharing policies among agents, and performing attention on sub-groups (of agents).While the Rover-Tower task has a lot of agents, each agent only gets information about its paired agent -in other words, the task itself has an intrinsically smaller number of "other agents" (conditioned on each agent) than the CTC environment. As a future direction, we are creating more complicated environments where each agent needs to cope with a large group of agents where selective attention is needed. This naturally models real-life scenarios that multiple agents are organized in clusters/sub-societies (school, work, family, etc) where the agent needs to interact with a small number of agents from many groups. We anticipate that in such complicated scenarios, our approach, combined with some advantages exhibited by other approaches would do well. We propose an algorithm for training decentralized policies in multi-agent settings. The key idea is to utilize attention in order to select relevant information for estimating critics. We analyze the performance of the proposed approach with respect to the number of agents, different configurations of rewards, and the span of relevant observational information. Empirical are promising and we intend to extend to highly complicated and dynamic environments. for j = 1... num critic updates do Update target critic and policy parameters: DISPLAYFORM0 T update ← 0 Update critic: DISPLAYFORM1 2, where Update policies: DISPLAYFORM2 DISPLAYFORM3 38: end function We train using Soft Actor-Critic BID10, an off-policy, actor-critic method for maximum entropy reinforcement learning. Our training procedure consists of performing 12 parallel rollouts, and adding a tuple of (o t, a t, r t, o t+1) 1...N to a replay buffer (with maximum length 1e6) for each timepoint. We reset each environment after every 100 steps (an episode). After 100 steps (across all rollouts), we perform 4 updates for the attention critic and for all policies. For each update we sample minibatches of 1024 timepoints from the replay buffer and then perform gradient descent on the Q-function loss objective FORMULA11, as well as the policy objective, using Adam BID15 as the optimizer for both with a learning rate of 0.001. These updates can be computed efficiently in parallel (across agents) using a GPU. After the updates are complete, we update the parametersψ of our target critic Qψ to move toward our learned critic's parameters, ψ, as in Lillicrap et al. FORMULA2; BID10:ψ = (1 − τ)ψ + τ ψ, where τ is the update rate (set to 0.002 for attention parameters and 0.04 for all other parameters). Using a target critic has been shown to stabilize the use of experience replay for off-policy reinforcement learning with neural network function approximators BID17. We update the parameters of the target policies,θ in the same manner. We use a discount factor, γ, of 0.99. All networks (separate policies and contained within the centralized critics) use a hidden dimension of 128 and Leaky Rectified Linear Units as the nonlinearity. We use 0.2 as our temperature setting for Soft Actor-Critic. Additionally, we typically use 4 attention heads in our attention critics unless otherwise specified. In order to compare to DDPG and MADDPG in our environments with discrete action spaces, we must make a slight modification to the basic algorithm. This modification is first suggested by BID20 in order to enable policies that output discrete communication messages. Consider the original DDPG policy gradient which takes advantage of the fact that you can easily calculate the gradient of the output of a deterministic policy with respect to its parameters. DISPLAYFORM0 Rather than policies that deterministically output an action from within a continuous action space, we use policies that produce differentiable samples through a Gumbel-Softmax distribution BID13. Using differentiable samples allows us to use the gradient of expected returns to train policies without using the log derivative trick, just as in DDPG. a∼π(s) [∇ a Q(s, a)∇ θ a] DISPLAYFORM1 In order to understand how the use of attention evolves over the course of training, we examine the "entropy" of the attention weights for each agent for each of the four attention heads that we use in both tasks (Figures 4 and 5). The black bars indicate the maximum possible entropy (i.e. uniform attention across all agents). Lower entropy indicates that the head is focusing on specific agents, with an entropy of 0 indicating attention focusing on one agent. In Rover-Tower, we plot the attention entropy for each rover. Interestingly, each agent appears to use a different combination of the four heads, but their use is not mutually exclusive, indicating that the inclusion of separate attention heads for each agent is not necessary. This differential use of attention heads is sensible due to the nature of rewards in this environment (i.e. individualized rewards). In the case of Treasure Collection, we find that all agents use the attention heads similarly, which is unsurprising considering that rewards are shared in that environment. In order to inspect how the attention mechanism is working on a more fine-grained level, we visualize the attention weights for one of the rovers in Rover-Tower (Figure 6), from the head that the agent appears to use the most (determined by looking at Figure 4), while changing the tower that said rover is paired to. In these plots, we ignore the weights over other rovers for simplicity since these are always near zero. We find that the rover learns to strongly attend to the tower that it is paired with, without any explicit supervision signal to do so. The model implicitly learns which Figure 5: Attention "entropy" for each head over the course of training for two collectors in the Treasure Collection Environment agent is most relevant to estimating the rover's expecture future returns, and said agent can change dynamically without affecting the performance of the algorithm. In order to test our model's ability to handle continuous action spaces, we add a network for each agent to learn a state-value function V i (o, a \i), which uses the same weighted attention embedding over other agents as Q i (o, a). The loss functions to learn both networks are provided by BID10. We test on an environment introduced in BID20 called Cooperative Navigation and compare to MADDPG. Our are presented in TAB3. This task does not require attention, as all agents are relevant to each others rewards at each time step. As such, it is unsurprising that our Figure 6: Attention weights when subjected to different Tower pairings for Rover 1 in Rover-Tower environment approach matches but does not surpass the performance of MADDPG. It is notable, however, both that attention does not harm performance in simple cases and that our approach handles continuous action spaces as well."

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: Recent research has intensively revealed the vulnerability of deep neural networks, especially for convolutional neural networks (CNNs) on the task of image recognition, through creating adversarial samples which `"slightly" differ from legitimate samples. This vulnerability indicates that these powerful models are sensitive to specific perturbations and cannot filter out these adversarial perturbations. In this work, we propose a quantization-based method which enables a CNN to filter out adversarial perturbations effectively. Notably, different from prior work on input quantization, we apply the quantization in the intermediate layers of a CNN. Our approach is naturally aligned with the clustering of the coarse-grained semantic information learned by a CNN. Furthermore, to compensate for the loss of information which is inevitably caused by the quantization, we propose the multi-head quantization, where we project data points to different sub-spaces and perform quantization within each sub-space. We enclose our design in a quantization layer named as the Q-Layer. The obtained on MNIST and Fashion-MNSIT datasets demonstrate that only adding one Q-Layer into a CNN could significantly improve its robustness against both white-box and black-box attacks. In recent years, along with the massive success of deep neural networks (DNNs) witnessed in many research fields, we have also observed their impressive failures when confronted with adversarial examples, especially for image recognition tasks. Prior work has demonstrated that an adversarial image can be easily synthesized by adding to a legitimate image a specifically crafted perturbation, which is typically imperceptible for human visual inspection. The generated adversarial image, however, is strikingly effective for causing convolutional neural network (CNN) classifiers to make extreme confident misclassification . This vulnerability of DNNs has stimulated the unceasing arms race between research on both attacking (; ; ; ; ;) and defending (; Samangouei et al. (2018b);; ) these powerful models. Among much existing work and a large variety of defense methods, several prior studies (; ;) have spent concerted efforts on defending adversarial attacks through input quantization. The principle idea of these methods is to use quantization to filter out small-scale adversarial perturbations. Recall that in prior work (; ;), it has been shown that the shallow layers of a CNN mostly capture fine-grained features including lines and curves. In the meantime, deeper layers learn coarse-grained yet semantically more critical features, which essentially discriminate different samples. Especially for classification tasks, it is natural to expect samples with the same classification label to share similar semantic information. As such, the semantic similarity between samples may be better revealed if we attend to their latent features learned by the intermediate layers of a CNN. Here we hypothesize that data points with similar semantic information should be distributed densely in the latent feature space. Thus, in order to more effectively filter out adversarial perturbations, we propose an alternative approach which quantizes the data representations embedded in the feature space produced by the intermediate layers of CNN classifiers. Interestingly, there have been other studies that develop similar approaches but for different purposes. For example, Wang et al. (2017; have applied k-means clustering on the intermediate feature maps of CNN models to discover explainable visual concepts. Recent methods, including VQ-VAE (van den) and SOM-VAE , were proposed to construct generative models for images and time-series data with discrete latent representations, which offer better explainability. However, to the best of our knowledge, the approach of applying intermediate layer quantization for CNN models has not been explored in the context of defending adversarial examples. In this work, we propose a quantization method that is realized by an extra intermediate layer, i.e., the quantization layer (Q-Layer). Our Q-Layer can be easily integrated into any existing architecture of CNN models. Specifically, the Q-Layer splits the mainstream of information that flows forward in a regular CNN model into two separate flows. Both flows share the same information passed by layers before the Q-Layer, but differ in the subsequent networks after the Q-Layer. These two flows produce two outputs, one is the quantized output, and the other is the Non-quantized output. Specifically, the non-quantized path is introduced to facilitate the gradient-based training, and to regularize the quantization operation. In the quantized path, we introduce non-differentiability to defend gradient-based attacks. It is important to note that, while gradient-based attacks cannot be directly applied to the quantized network, they can still be conducted by following the nonquantized path. Also, similar to most input transformation methods proposed for defending adversarial examples, our quantization will inevitably lose some feature information, which might be useful for classification. In order to compensate for this loss of information, we further propose multi-head quantization, where we project data points to different sub-spaces and perform quantization within each sub-space. In particular, we perform the projection by re-weighting the input-channels of CNN with trainable parameters. This projection process can be interpreted as performing feature extraction from different points of view, hence help retain the overall effectiveness of our method without causing much performance degradation for the model to be protected. Last but not least, our proposed method can be readily combined with other existing defenses, e.g., adversarial training , to jointly improve the adversarial robustness of a protected CNN classifier. In summary, we make the following contribution: • We propose a quantization-based defense method for the adversarial example problem by designing a quantization Layer (Q-Layer) which can be integrated into existing architectures of CNN models. Our implementation is online available 1. • We propose multi-head quantization to compensate for the possible information loss caused by the quantization process, and bring significant improvement to the adversarial robustness of an armed model under large perturbation. • We evaluate our method under several representative attacks on MNIST and Fashion-MNIST datasets. Our experiment demonstrate that the adoption of the Q-Layer can significantly enhance the robustness of a CNN against both black-box and white-box attack, and the robustness can be further improved by combining our method with adversarial training. 2 RELATED WORK 2.1 ADVERSARIAL ATTACK Given a neural network classifier N with parameters denoted by w, N can be regarded as a function that takes an input x ∈ R dx and produces an classification label y, i.e., N (x; w) = y or N (x) = y for notation simplicity. In principle, the goal of the adversarial attack is to create a perturbation δ ∈ R dx to be added to a legitimate sample x for creating an adversarial example, i.e., x + δ, which causes the target model N to produce a wrong classification . Depending on different threat models, adversarial attacks are categorized as black-box attacks or white-box attacks . Specifically, it is commonly assumed in the white-box attack scenario, that an attacker knows every detail of the target model. This dramatically eases the generation of impactful adversarial examples, and has stimulated researchers to propose various white-box attack methods, including the fast gradient sign method (FGSM) , the basic iterative method (BIM) , the Carlini-Wagner (CW) attack , and DeepFool . On the contrary, in the black-box attack scenario, an attacker is typically assumed to be restricted for accessing detailed information, e.g., the architecture, values of parameters, training datasets, of the target model. There have been many black-box attack methods proposed in prior work (; ;). Representative black-box attacks typically exploit the transferability of the adversarial examples, hence is also referred to as transfer black-box attacks. Explicitly, in transfer black-box attacks, an attacker can train and maintain a substitute model, then conduct white-box attacks on the substitute model to generate adversarial samples which retain a certain level of attack power to the target model. Since both black-box and white-box attacks rely on the white-box assumption, in the following, we mainly introduce several representative white-box attacks, namely the FGSM, BIM and CW attacks, which are also employed in our experiments due to their wide adoption as the benchmark attack methods (Samangouei et al. (2018a; b) ). Fast gradient sign method (FGSM) proposed FGSM, in which δ is calculated by scaling the l ∞ norm of the gradient of the loss function L with respect to a legitimate input x as follows: where represents the maximally allowed scale of perturbation. This method represents a one-step approximation for the direction in the input space that affects the loss function most significantly. Basic iterative method (BIM) proposed the BIM attack, which iteratively performs the FGSM hence generates more impactful adversarial examples at the expense of computational efficiency. Carlini-Wagner (CW) attack aimed to find the smallest perturbation to fool the target model, by solving the following optimization problem: where c > 0 is a tunable positive constant and p represents different norms. In our experiment, we consider l ∞ norm. L is designed to satisfy that L(x, δ) < 0 if and only if N (x + δ) = N (x). There have been many different defense methods (; ; ; Samangouei et al. (2018a; b);; ) developed to battle with a large body of attacking methods. Here we briefly introduce three representative ones. Adversarial training In the same work that introduced also proposed to train a target model with an augmented training set, which contains both original samples and adversarial samples generated by the target model itself. Considering the low computational cost needed by FGSM, it is usually used for generating target-specific adversarial examples in adversarial training. introduced a more general framework for generating adversarial examples and proposed to use iterative attacks, such as BIM, to produce adversarial samples for augmenting the training set. The ing defense is regarded as among the most effective defense methods . Input quantization proposed feature squeezing, which transforms the input by reducing the range of pixel values then employs a filter to smooth the input image. propose to encode the input with one-hot encoding or thermometer coding. add Gaussian noise to the input and cluster input pixels to perform quantization. Input purification Samangouei et al. (2018b) propose PixelDefend, where they used a PixelCNN (van den) to model the distribution of input pixels, and differentiate adversarial samples from legitimate samples. Their show that the pixel distribution of adversarial samples is indeed "unnatural" in comparison with legitimate samples. Then they proposed to purify input pixels to obtain a natural distribution and pass the purified input to the target model for further processing. Given a CNN which has the Q-Layer injected into the intermediate part (e.g., after the convolutional block and before the fully connected block or between two convolutional blocks) of the network, its input goes through the quantization path (Q-path), which consists of several steps of processing, including projection, quantization, and concatenation, then produce the quantized output for the subsequent network, as depicted in Figure 1. Note, that the quantization step forces pre-quantization data representation (outputs of the projection step), i.e., z e,j (j = 1, 2, 3), to be replaced by the post-quantization representation, i.e., z q,i, (i = 1, 2, 3). Since the quantization step introduces nondifferentiability in the Q-path, the backpropagation operation cannot be directly applied for this path. In order to make the enhanced model still trainable, we propose to concatenate the multihead projection and pass the concatenation to a separate subsequent network along the nonquantization path (E-path). It is natural to consider about having both paths connected to subsequent networks that are identical by sharing their weights, whereas, in our experiments, we observed inferior performance in comparison with having both paths connected to networks with different weights. Interested readers could refer to Appendix G for an ablation study which demonstrates this effect. Correspondingly, in Figure 1, we illustrate four backward paths that jointly contribute to updating the model. In particular, path 1, 3, and 4 constitute the backward path for the quantization operation, and path 2 represents the backward path for the non-quantization step. Given a neural network N, we can split it into two sub-network, denoted by N F and N B respectively, from somewhere in the middle of the network. Specifically, in this work, we split a regular CNN into an image encoder network which contains convolutional blocks and a fully-connected classifier network. Then given an input x, the output y of such a neural network is calculated by: We further assume the size of the intermediate output, i.e., z e = N F (x), as n×w ×h×d, where n is the batch size, w, h are the width and height of the intermediate output, d is the number of channels. We consider z e to be composed by n * w * h panels, where an arbitrary panel j, i.e., z j e, is a vector of length d. As shown in Figure 1, the Q-Layer takes z e as input panels, and outputs quantized panels After identifying the closest concept Q i * for z j e, we model the quantization operation as multiplying the entire concept matrix Q by an n c -dimensional one-hot encoded identification vector I i *. As previously mentioned, we pass z q and z e to two different subsequent networks, denoted by N Q and N E respectively. Accordingly, we refer y q and y e as the outputs produced by the Q-path and the E-path, respectively. Then given an input x, the final output is as follows: 3.3 MULTI-HEAD QUANTIZATION Our quantization step introduced above essentially describes a built-in online clustering process. However, clustering in high dimensional space is challenging, mainly due to the existence of many irrelevant dimensions . Furthermore, in order to find rich concepts that reflect the similarity and difference between samples, our quantization step needs to locate clusters that may be embedded in different sub-spaces. As such, we introduce sub-space clustering into our design and propose multi-head quantization to address these challenges as well as alleviate the possible information loss caused by the single-head quantization step. More specifically, we first project an input representation to K sub-spaces, where K is a pre-defined hyper-parameter. Then we perform single-head quantization within each sub-space. At last, we concatenate the quantization produced in all sub-spaces to obtain the final output. In the projection step, we apply a re-weight mechanism to d channels of an input representation. As mentioned before, we can consider a input batch as a (n×w×h) panels, where a panel is represented as a d-dimensional vector containing features generated by d channels. Given the j-th panel z j e ∈ R 1×d, let z j e,i be the i-th sub-space projection and W i ∈ R d×d, b i ∈ R 1×d be the projection parameters of the i-th sub-space, then we have: where denotes the Hadamard product. Denote the projection operation conducted in the i-th subspace by P i and the corresponding concept matrix in this sub-space by Q i. Then the operation of concatenating the outputs from all sub-spaces is represented as follows: Given an input x, the formulations for y q, y e in the case of the multi-head quantization is as follows: By following the design of VQ-VAE (van den) and SOM-VAE , we decompose the quantization loss into two separate loss terms and finally specify the following four terms (as shown in Figure 1) in the final training loss, i.e., where c 1, c 2, α, β are tunable hyper-parameters. Specifically, the first loss L Q represents the standard cross-entropy loss caused by the quantized output. This loss is specified to optimize the weight parameters of N Q and the Q-Layer to fit the classification task. As previously mentioned, the Qpath does not allow direct backpropagation, as such, in order to optimize N F, we set the second cross-entropy loss, i.e., L E, to train N F and N E by following the E-path. The last two loss terms are specified for optimizing the quantization performance by directly evaluating the distance between concept vectors, which can be regarded as clustering centroids, and learned data projections. In particular, the third loss L Q→E measures the l 2 distance between z q and a fixed z e. In other words, we apply the "stop-gradient" operation (denoted as sg) to z e to ensure that this loss only moves z q to z e instead of the opposite. This is a direct analogy to clustering as by minimizing L Q→E, all row vectors of the parameter matrix Q keep moving towards different clustering centers. Similarly, the fourth loss, i.e., L E→Q, measures the "commitment" of data projections to their corresponding clustering centroids. Here we apply the stop-gradient operation to z q while minimizing this loss. By optimizing L E→Q, we try to shape the learned projections of samples to be more similar to their corresponding concepts, hence eventually forming densely distributed clusters. Although we have followed prior work (van den) to decompose the quantization loss as described above and allowed more flexible control over L Q→E and L E→Q by tuning α and β, however, in our experiments, after tuning α and β with cross-validation, we observed the ratio of these two hyper-parameters, if within certain range (for example, 0.1 to 10), had insignificant influence on the final performance. A similar effect has also been reported by van den. As such, in Section 4, we mainly present the obtained by setting α = β. If the concept matrix is not properly initialized or optimized, a concept which is far away from all data projections may remain constant during training. We refer to concepts that behave as described above as "inactive" concepts and refer to other concepts that can be effectively updated as "active" concepts. This issue may severely impact the final performance when there are relatively many inactive concepts. To update inactive concepts, we design two updating strategies. One is to force an inactive concept to move its closest panel directly. The other is to re-initialize an inactive concept as its closest panel. In our experiments, we mainly use the first strategy due to its better effectiveness. Specifically, the first strategy is implemented by adding a special loss term, which calculates the distance between each inactive concept to its closest panel. We then optimize this loss after each training epoch. In our experiments, we considered two types of attacks, namely the black-box and the white-box attacks. Under each type of attacks, we evaluated the effectiveness of our proposed Q-Layer by comparing both the accuracy and robustness of two standard CNNs built with and without the Q-Layer obtained on a clean testing set and an adversarial testing set constructed from this clean testing set. The accuracy achieved by a CNN on the clean testing set reflects its standard generalization performance, while its robustness is represented by the accuracy obtained by this CNN on the adversarial testing set. In particular, under white-box attacks, for a CNN equipped with the Q-layer, the direct gradient-based attack can only be applied to the E-path, as mentioned previously. We refer to this type of attack as the E-path white-box attack. Furthermore, we assumed that an attacker might attack the non-differentiable Q-path by ignoring the quantization operation and directly concatenate z e to N Q to build a substitute attack path following equation 9, which we refer as the Q-path white-box attack. Moreover, under both types of attacks, we also evaluated CNNs built with and without the Q-Layer by considering the case where both models were enhanced by adversarial training. The models considered in this case are referred to as adversarial-trained models. Correspondingly, the models trained without adversarial training are referred to as raw-trained models. Note, in the following experiments, we only focus on the classification accuracy obtained by the Q-path by CNNs armed by the Q-Layer. We report the accuracy obtained by the E-path in Appendix D. The datasets adopted in our experiments include the MNIST and the Fashion-MNIST datasets 2. For each dataset, we had five subsets of samples, including a training set, an adversarial training set, a validation set, a clean testing set, and an adversarial testing set. A clean validation set was first constructed by randomly selecting 10,000 samples from the original training set, hence leaving the final training set contained the rest 50,000 training samples. Then we created adversarial examples from this clean validation set by using a pre-trained CNN A 3 and the FGSM method. This pre-trained CNN A only acted as a source model for creating adversarial validation samples and was not involved in either the training or testing process. The clean validation set was then mixed with its adversarial counterpart to build the final validation set. This mixed validation set was used for selecting models that achieved the best performance. Through this, we attempted to avoid the case of selecting models that can only perform well with clean samples. Instead, we tried to evaluate the performance of models that are naturally robust against adversarial examples. A comparison between the obtained following the above scheme and the obtained by models selected by using the clean validation set is provided in Appendix C. During the testing phase, depending on different attack scenarios, we constructed different adversarial testing sets. Specifically, under black-box attacks, we independently trained two source models for generating two sets of adversarial testing samples. One of the source models had the same structure as the CNN A but was trained with a different random initialization. The other source model, denoted by CNN B, had a structure that was different from that of CNN A. Both source models were used to simulate the scenario where an attacker generated adversarial examples from some unknown models. Under the white-box attacks, for each target model, its associated adversarial testing set was generated by directly attacking this model. Recall that we can apply our method in tandem with adversarial training. Therefore, on the MNIST, we created adversarial examples for adversarial training by setting = 0.2. For the mixed validation and adversarial testing set, we created adversarial examples by setting = 0.2/0.3. This setup is due to the observation that FGSM-based adversarial training is less effective against adversarial samples created with a larger scale of perturbation. As a , we attempted to simulate a more critical scenario in this setup. On the Fashion-MNIST, we noticed that by setting = 0.2/0.3, the ing adversarial images were severely distorted. As such, we created adversarial examples by setting = 0.1/0.2. In our experiments, we compared the robustness of three target models: CNN A, CNN with a basic Q-Layer (K = 1, n c = 64, denoted as Q-base), and CNN with a large Q-Layer (K = 4, n c = 64, denoted as Q-large). Q-base and Q-large share the same architecture with CNN A, but with an additional Q-Layer after the convolution blocks and before the fully-connect layer. We have also evaluated the case where the Q-Layer was inserted between two convolution blocks of CNN A. The are showed in Section 4.4. Besides, we provided the architectures of the target and substitute models, the hyper-parameters, and other implementation details in Appendix A/B. Table 1: Comparison of classification accuracy for different target models on the MNIST under black-box attack with source model CNN A. We use "N +adv" to refer adversarial-trained models. As previously mentioned in Section 4.1.1, we use two substitute models, i.e., CNN A and CNN B, to implement black-box attacks for a target model. Notably, in our experiments, we observed that the attacks conducted by using CNN A were generally more impactful than the attacks conducted by using CNN B. As such, due to space limitations, here we only demonstrate the obtained under attacks performed by CNN A and provide the obtained for CNN B in Appendix E. Also, we have adopted three representative attacks, namely the FGSM attack, the BIM attack, and the CW attack (introduced in Section 2). In Table 1a and Table 1b, we only present the obtained under the FGSM and BIM attacks, as we have observed that, despite being a powerful attack under the white-box scenario, the CW attack produced less effective black-box attacks. MNIST From Table 1a, we can observe that, by inserting the Q-Layer into the CNN, its robustness is improved significantly. Take the for raw-trained Q-large as an example, in comparison with the unarmed CNN, the accuracy under the FGSM attack of = 0.3 rises from 40.03 to 65.19, and the accuracy under BIM attack of = 0.3 rises from 16.02 to 67.01, which is even higher than the accuracy obtained by the adversarial-trained CNN. With adversarial training, the robustness of all models increases. In particular, while these models have similar robustness under small-scale perturbations, as the scale of perturbation increases, the robustness of the target CNN decreases more significantly than Q-large and Q-base, especially for the BIM attack. Table 1b 4, we show the obtained under black-box attacks for Fashion-MNIST. Similar to MNIST, we can observe improved robustness for raw-trained Q-base and Qlarge, in comparison with CNN. Also, after adversarial training, the robustness of the three models are close under small perturbation attack, while Q-base and Q-large outperform CNN under large perturbation. In addition to the shown above, we have also performed several sets of ablation study for exploring the influence of the number of sub-spaces and the number of concepts on black-box robustness and presented the in Appendix F. Visualizing the distributions of pixels and concepts Inspired by PixelDefend (Samangouei et al. (2018b) ), which proposed to use a PixelCNN van den to model and compare the distribution of pixels in clean and adversarial samples, here we use a PixelCNN to model and compare the distribution of concepts learned from clean and adversarial samples. As shown in Figure 2a /2b, for the MNIST dataset and under the FGSM attack, the distribution of pixels exhibits much more significant distortion than the distribution of learned concepts. This clearly indicates that the quantization effectively blocks the adversarial perturbations, hence ing in improved robustness. Table 2: Comparison of classification accuracy for different target models on the MNIST under white-box attack. As mentioned in Section 4.1, in order to conduct attacks even when the Q-path does not suffer from gradient-based attacks, we assume that an attacker might use a shortcut from z e to N Q to perform attacks. However, we observed in our experiments (shown in Appendix H) that this type of attack was generally weaker than the E-path attack. Therefore, here we only present obtained by the E-path attack in Table 2a and Table 2b. As shown in Table 2a, raw-trained Q-large and Q-base significantly outperform CNN. For attacks with small perturbations, adversarial trained CNN performs slightly better than Q-large and Q-base. However, as the scale of perturbation increases, Q-large and Q-base restore their superior robustness. Similar can also be observed for Fashion-MNIST, as shown in Table 2b. In Section 4.4, we also show that inserting the Q-Layer between two convolution blocks brings even stronger robustness against white-box attack. In the following experiments, we show the flexibility of varying the position where the Q-Layer is inserted. In particular, we refer to a CNN which has a Q-Layer inserted between its two convolution blocks as Q-inner. The Q-inner model has the same parameter setting as the Q-large (K = 4, n c = 64) model used in the previous experiments. Table 3: Comparison of classification accuracy for Q-large and Q-inner on the MNIST, under blackbox attack and white-box attacks. In Table 3a and Table 3b, we demonstrate the comparison under both black-box and whitebox attacks on the MNIST dataset. Under black-box attacks, we observe that Q-inner and Q-large achieve comparable robustness. However, under the white-box attacks, it can be seen that Q-inner performs much better than Q-large. These not only demonstrate the flexibility of the Q-Layer, but also indicate that applying the quantization at an early stage of the information propagation can be more effective for filtering out an adversarial perturbation, of which the scale has not been sufficiently amplified as the perturbation propagates to the deeper parts of the network. In this paper, we have designed and implemented a quantization layer (Q-Layer) to protection CNN classifiers from the adversarial attacks, and presented the experiment which show that, by simply inserting one Q-Layer into a regular CNN, its adversarial robustness under both white-box and black-box attacks obtains significant improvement. Moreover, we have combined our method in tandem with adversarial training. The empirical show that the Q-layer can make a CNN benefit more from adversarial training and even perform well under attacks with larger perturbations. One limitation of this work is due to the uncertainty introduced by the random initialization of concept matrix. This issue also exists in many other clustering algorithms. In this work, we alleviate the impact of this issue by reactivating inactivate concepts. Future work would pursue other approaches on constructing the concept matrix, e.g., regularizing the concept matrix with specific semantic constrains, and using the E-path as a learned index to retrieve information stored in the concept matrix, which acts as an external memory. Model A,B are two neural network architectures used in Samangouei et al. (2018a) for the MNIST and Fashion-MNIST datasets. In this paper, A is used for both target CNN and substitute model. B is only used for substitute model. When train CNN with a Q-Layer, we use RAdamOptimizer and set learning rate as 0.001. As for losses, we set c 1 = 1, c 2 = 1, α = 0.001, β = 0.001. The scale of α and β are chosen to make α · L Q→E and β · L E→Q have smaller but close magnitude of L Q and L E. The ratio of c 1 and c 2 do not have significant influence on the , for they are optimizing different part of the network. In our experiments, We didn't find significant change of when we tuned the ratio of α and β. As for inactive concepts updating strategy one, i.e., optimizing the distance between inactive concepts to its closest data point, we use AdamOptimizer and set learning rate as 0.001. Specifically, when training Q-inner on MNIST, we set α = 0.0001, β = 0.0001 and use the second inactive concept updating strategy (otherwise it could not converge), keeping other hyperparameters unchanged. All the hyper-parameter tuning are based on train set and validation set. We train each model three times with different random initialization and select the model with highest validation accuracy as the final model to test. We use Adversarial Robustness 360 Toolbox to generate adversarial samples. For each attack, we adjust, set batch size to be 128, leaving other hyper-paramters as default settings in Adversarial Robustness 360 Toolbox. Additionally, when implementing BIM attack, we set step to be 0.01. In this section, we compare two CNN models on MNIST, all share the same structure with CNN A, independently trained, but are selected on the clean validation set and mixed validation set, respectively. As shown in Table 5, by selecting models on the mixed validation set, we get a much more robust model, with slight sacrificing on clean accuracy. In this section, take MNIST as an example, we compare the Q-path classification accuracy and Epath classification accuracy under black-box attack and while-box attack. In the following, we've reported two classification accuracy scores for each of the Q-base and Q-large models. One score represents the accuracy obtained by following the Q-path, and the other represents the accuracy obtained by following the E-path. As shown in Table 6a and Table 6b, for both raw-trained Q-base and Q-large, the accuracy obtained by the Q-path is significantly higher than that obtained by the E-path, especially when is large, which prove the efficiency of quantization. We also note that in certain cases, after adversarial training, for Q-large and Q-base under small perturbations, the accuracy obtained by the Q-path becomes slightly lower than that obtained by the E-path. Recall that the E-path represents the passage in a regular CNN. Therefore, this difference between the Q-path and E-path may be due to the reason that the quantization inevitably causes loss of information, hence leading to a lower accuracy score for the Q-path. In this appendix, we compare the classification of different target models on MNIST and Fashion-MNIST, under black-box attack with source model CNN B, whose architecture is different from the target CNN. The on MNIST is showed in Table 7a. The on Fashion-MNIST is showed in Table 7b. Same as the observation in Table 1a and Table 1b, inserting a Q-Layer improves the robustness of the network. To further study the effect of K (the number of sub-spaces) and n c (the number of concepts), we insert Q-layer with K = 1, 2, 4 and n c = 16, 64, 128 to CNN and compare its robustness on MNIST under FGSM black-box attack with = 0.3. As shown in Table 8a, before adversarial training, more sub-spaces bring stronger robustness. After adversarial training, when n c = 16, models with larger K demonstrate higher accuracy; however, when n c = 64/128, the relationship of K and accuracy is not clear. We speculate that when n c = 16, quantizing features causes too much loss in information, therefore additional information introduced by adding more sub-spaces significantly improves accuracy. However, when n c is large enough to represent different concepts, the importance of K decreases after adversarial training. In the shared subsequent network setting, we let N Q and N E share parameters, while in the separated subsequent network setting, N Q and N E do not. We compare the robustness of two independent Qlarge models with separated subsequent networks and with shared subsequent network. We report the black-box attack on MNIST in Table 9 and white-box attack in Table 10. The proves that, the separated subsequent networks do help robustness under both black-box attack and white-box attack. Nonetheless, separated subsequent networks are harder to optimize, thus we recommend users to use separated subsequent networks if they are not very deep, and use shared subsequent networks otherwise. Table 10: Classification accuracy comparison of Q-large with separated subsequent networks and shared subsequent networks on MNIST, under white-box attack. H WHITE-BOX ATTACK ON Q-PATH As mentioned before, the original Q-path for Q-layer could not be attacked with gradient-based method. A clever attacker might build a shortcut path from z e to N Q to attack, which we refer as Q-path white-box attack. However, due to the differences between N Q and N E, attack through short-cut path is usually weak. In this section, we show the Q-path white-box attack and E-path white-box attack in Table 11. We could observe that, at most of the time, attacking Q-path reduces less accuracy than attack E-path. Table 11: Classification accuracy comparison of different target models on MNIST, under Q-path and E-path white-box attack. We underline the when Q-path attack is stronger than E-path attack.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a quantization-based method which regularizes a CNN's learned representations to be automatically aligned with trainable concept matrix hence effectively filtering out adversarial perturbations.

Represent the short summary for finding the corresponding scientific article (long)

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.