Question: In cooperative multi-agent reinforcement learning (MARL), how to design a suitable reward signal to accelerate learning and stabilize convergence is a critical problem. The global reward signal assigns the same global reward to all agents without distinguishing their contributions, while the local reward signal provides different local rewards to each agent based solely on individual behavior. Both of the two reward assignment approaches have some shortcomings: the former might encourage lazy agents, while the latter might produce selfish agents. In this paper, we study reward design problem in cooperative MARL based on packet routing environments. Firstly, we show that the above two reward signals are prone to produce suboptimal policies. Then, inspired by some observations and considerations, we design some mixed reward signals, which are off-the-shelf to learn better policies. Finally, we turn the mixed reward signals into the adaptive counterparts, which achieve best in our experiments. Other reward signals are also discussed in this paper. As reward design is a very fundamental problem in RL and especially in MARL, we hope that MARL researchers can rethink the rewards used in their systems. In reinforcement learning (RL), the goal of the agent is formalized in terms of a special signal, i.e., reward, coming from the environment. The agent tries to maximize the total amount of reward it receives in the long run. Formally, we express this idea as the Reward Hypothesis BID31: the goal of RL agent can be exactly described as the maximization of the expected value of the cumulative sum of a received scalar reward signal. It is thus critical that the rewards truly indicate what we want to accomplish. One reward design principle is that the rewards must reflect what the goal is, instead of how to achieve the goal 1. For example, in AlphaGo, the agent is only rewarded for actually winning. If we also reward the agent for achieving subgoals such as taking its opponent's pieces, the agent might find a way to achieve them even at the cost of losing the game. A similar example of faulty reward function is provided by BID26: if we reward the action of cleaning up dirt, the optimal policy causes the robot to repeatedly dump and clean up the same dirt. In fact, the how reward encodes human experience, which is heuristic in some extent. Based on the heuristic how reward, it is really easy to deviate from the ultimate goal. However, as BID35 point out, the exact what reward that encodes the performance objective might be awful to use as a training objective. It will in slow and unstable learning occasionally. At the same time, a training objective that differs from the performance objective can still do well with respect to it. For example, the Intrinsically Motivated Reinforcement Learning (IMRL) BID6 BID27 ) combines a domain-specific intrinsic reward with the reward coming from the environment to improve learning especially in sparse-reward domains. Although reward design problem in single-agent RL is relatively tractable, it becomes more thorny in multi-agent reinforcement learning (MARL), as MARL is naturally more complex than singleagent RL. As we know, the global reward and local reward have long been proved to be defective: the former might encourage lazy agents, while the latter might produce selfish agents. Inspired by the success of intrinsic reward in single-agent RL, we hypothesize that similar methods may be useful in MARL too. Naturally, in this paper, we ask and try to answer a question:Can we formulate some special rewards (such as intrinsic reward) based on the meta what rewards to accelerate learning and stabilize convergence of MARL systems? Specifically, in this paper, we propose several new MARL environments modified from the well known Packet Routing Domain. In those environments, the goal is to figure out some good flow splitting policies for all routers (i.e., agents) to minimize the maximum link utilization ratio in the whole network. We set the meta reward signals as 1 -max(U l). We argue that the meta reward signals are some kinds of what rewards because they tell the agents that we want to minimize max(U l), i.e., minimize the maximum of all link utilization ratios. For detailed discussions, we refer the readers to the proposed environments and rewards in Section 3 and 4.Based on those environments and the meta what rewards, we can focus on our reward design research purpose. Specifically, we firstly show that both of the widely adopted global and local reward signals are prone to produce suboptimal policies. Then, inspired by some observations and considerations, we design some mixed reward signals, which are off-the-shelf to learn better policies. Finally, we turn the mixed reward signals into the adaptive counterparts, which achieve best in our experiments. Besides, we also discuss other reward signals in this paper. In summary, our contributions are two-fold. We propose some new MARL environments to advance the study of MARL. As many applications in the real world can be modeled using similar methods, we expect that other fields can also benefit from this work. We propose and evaluate several reward signals in these MARL environments. Our studies generalize the following thesis BID6 BID35 in single-agent RL to MARL: agents can learn better policies even when the training objective differs from the performance objective. This remind us to be careful to design the rewards, as they are really important to guide the agent behavior. The rest of this paper is organized as follows. Section 2 introduces briefly, followed by the proposed environments and rewards in Section 3 and 4, respectively. We then present the experiments and discussions in Section 5 and 6, respectively. Section 7 concludes this work. As reward is the foundation of RL, there are many related studies. We only introduce the most relevant fields of this work. Topics such as Inverse RL BID23 and Average Reward RL BID19 are not included. The only way to talk with your agents might be the reward, as expressed by the well known Reward Hypothesis .When considering reward design for single objective RL problem, we should always be aware of whether the designed reward is a kind of what reward rather than how reward. For multiple objectives RL problem, researchers have to design sub-rewards for different objectives, and the final reward is a weighted sum of those sub-rewards. Unfortunately, the weights have to be adjusted manually even in recent studies BID12 BID17 BID8.For single-agent RL, there are many remarkable reward design studies. The most relevant field may be the IMRL. A recent Deep RL work based on IMRL is VIME . It uses a surprise reward as the intrinsic reward to balance exploitation and exploration, and achieves better performance than heuristic exploration methods. Another successful model is Categorical DQN BID1, which considers the long time run reward, or value, used in approximate RL. The authors model the value distribution rather than the traditional expected value. They obtain anecdotal evidence demonstrating the importance of the value distribution in approximate RL. BID18 use the temporal logic (TL) quantitative semantics to translate TL formulas into realvalued rewards. They propose a temporal logic policy search (TLPS) method to solve specify tasks and verify its usefulness. However, the reward design studies for MARL is so limited. To the best of our knowledge, the first (and may be the only) reward design study about Deep MARL is the well known Independent DQN BID32. By setting different rewarding schemes of Pong, the authors demonstrate how competitive and collaborative behaviors emerge. Although the rewards used are very elaborate, there are only two agents in Pong, which limits its generalization ability. Credit assignment usually has two meanings. In single-agent RL, it is the problem that how much credit should the current reward be apportioned to previous actions. In MARL, it is the problem that how much credit should the reward obtained at a team level be apportioned to individual agents. In fact, credit assignment is a subclass of reward design, and the two fields can often be seen as the same in MARL. As far as we know, credit assignment approaches in MARL can be divided into two categories: traditional global/local reward and other rewards. The global reward approach assigns the same global reward to all agents without distinguishing their contributions. On one hand, the lazy agents often receive higher rewards than what they really contribute to, which leaves lazy agents with no desire to optimize their policies. On the other hand, the diligent agents can only receive lower rewards even when they generate good actions, as the lazy agents often make the whole system worse. This makes the diligent agents confuse about what is the optimal policy. The local reward approach provides different rewards to each agent based solely on its individual behavior. It discourages laziness. However, agents do not have any rational incentive to help each other, and selfish agents often develop greedy behaviors. Other reward signals need more complex computation or depend on non-universal assumptions. After having considered several other options, we finally choose the Packet Routing Domain as our experimental environments. The reasons are as follows.• Firstly, they are classical MARL environments. There are many researchers studying packet routing problem BID2 BID7 BID29 BID37 BID4 BID33 BID36 BID9 BID25 BID20 ).• Besides, many real-world applications can be modeled using similar methods, for example, the internet packet routing, electric power supply, natural gas transportation and traffic flow allocation. We expect that these fields can also benefit from this work.• And most importantly, the what reward of these environments can be easily figured out, so we can focus on our reward design research purpose. Specifically, in these environments, the goals are high throughput and low link utilization ratio. In the real world, we also assume that the whole network capacity is bigger than the total flow demands 2, so the throughput is equal to the flow demands if we can find good packet routing policies. With this assumption, we set the meta reward signals as 1 -max(U l). We argue that the meta rewards are some kinds of what rewards because they tell the agents that we want to minimize max(U l), i.e., minimize the maximum of all U l.• Another attractive feature of those environments is that when we compute the global reward signal, the local reward signals of each agent can also be calculated without too much additional computation. See the detailed discussions in Section 4.More concretely, we consider the environments as shown in Figure 1, 2 and 3. Currently, the Internet is made up of many ISP networks. In each ISP network, as shown in Figure 1, there are several edge routers. Two edge routers are combined as ingress-egress router pair (IE-pair). The i-th IE-pair has a input flow demand F i and K available paths that can be used to deliver the flow from ingress-router to egress-router. Each path P k i is made up of several links and each link can belong to several paths. The link L l has a flow transmission capacity C l and a link utilization ratio U l. As we know, high link utilization ratio is bad for dealing with bursty traffic, so we want to find some good flow splitting policies jointly for all IE-pairs across their available paths to minimize the maximum link utilization ratio in the ISP network. More formally, the goal is the same as BID16: DISPLAYFORM0 subject to the constraints: DISPLAYFORM1 where F i * y DISPLAYFORM2 Before the definition of the designed rewards, we firstly define some link utilization ratio sets.• ALL set of any router. It contains the link utilization ratios of ALL links in the network. For example, in Figure 1, DISPLAYFORM0 • DIRECT set of a given router. It only contains the link utilization ratios of the DIRECT links of this router. For example, in Figure 1, DIRECT (e) = {U ec, U ed, U ef}.• BASIN set of a given router. It only contains the link utilization ratios of the BASIN links of this router. And the BASIN links of a router are the links that this router can route flow on. For example, in Figure 1, DISPLAYFORM1 Now, we are ready to define the designed reward signals as follows. A simplified illustration of these reward signals is shown in FIG3.• Global Reward (gR). The reward is calculated based on ALL set: gR = 1 − max(ALL).• Direct Local Reward (dlR). The reward is calculated only based on DIRECT set of a given router. For example, dlR(e) = 1 − max(DIRECT (e)).• Basin Local Reward (blR). The reward is calculated only based on BASIN set of a given router. For example, blR(e) = 1 − max(BASIN (e)).• Direct Mixed Reward (dlgMixedR). The reward is the sum of gR and dlR. For example, dlgM ixedR(e) = gR + dlR(e).• Basin Mixed Reward (blgMixedR). The reward is the sum of gR and blR. For example, blgM ixedR(e) = gR + blR(e).• Direct Adaptive Reward (dlgAdaptR). The reward is the sum of gR and w*dlR, where w is the weight of dlR. We decay w gradually at the training procedure. For example, dlgAdaptR(e) = gR + w * dlR(e).• Basin Adaptive Reward (blgAdaptR). The reward is the sum of gR and w*blR, where w is the weight of blR. We decay w gradually at the training procedure. For example, blgAdaptR(e) = gR + w * blR(e).From the above definitions, we can see that gR is what reward. It tells the agents that we want to minimize the maximum of all U l. Besides, both dlR and blR can be seen as some kinds of what rewards from the local perspective of individual agent. Specially, both mixed rewards and adaptive rewards are simple combinations of those what rewards. Despite the simplicity of those designed rewards, we can focus on our research purpose: can those rewards accelerate learning and stabilize convergence of MARL systems? And as mentioned in previous sections, we can calculate these rewards at a low cost, because DIRECT set and BASIN set are subsets of ALL set. To make our contributions more clearly, please note that traditional packet routing studies often use gR and dlR, all other signal forms are firstly introduced in this paper as far as we know. In order to make the experimental environments consistent with the real-world systems, we highlight the following setting. More detailed information can be found in the Appendix Section A.1 and A.2.• The routers are partially observable as they are located in different places. We use the recent proposed ACCNet BID20 as the experimental method to handle this problem.• The time delay of router, link and reward signal cannot be neglected. That is to say, actions have long term effect on the environments, which makes the task more challenging.• Both synthetic flow and real flow trajectory from the American Abilene Network 3 are used to test the proposed rewards. For different rewards, we run 10 independent experiments on different topologies, and the averaged convergence rates (CR) and maximum link utilization ratios (U l *) are shown in TAB0. From the perspective of topology, we can see that the convergence rates of any reward signal are decreasing gradually as the topology becoming more complex, which is reasonable. From the perspective of reward, we draw the following . Firstly, both dlR and blR are better than gR, which means that the widely used gR in other MARL environments is not a good choice for the packet routing environments. The bad performance of gR motivates us to discover other more effective reward signals. Importantly, the proposed blR seems to have similar capacity with dlR, but when we consider mixed reward signals and adaptive reward signals, the differences between them are obvious. For example, blgM ixedR and blgAdaptR can achieve higher convergence rates than dlgM ixedR and dlgAdaptR on Simple Topology and Moderate Topology, while dlgM ixedR and dlgAdaptR has better performance on Complex Topology than blgM ixedR and blgAdaptR. In our opinion, dlR has low f actoredness but high learnability, while blR can better balance f actoredness and learnability 4, which makes dlR more suitable for symmetrical Complex Topology and blR more suitable for asymmetrical Simple Topology and Moderate Topology. Anyways, the proposed blR is very necessary for the packet routing environments. Besides, on the whole, the adaptive reward signals are better than the mixed reward signals, and both of them are better than the meta rewards (we refer to gR, dlR and blR). But please note that the mixed reward signals can achieve good without any change of the experimental setting 5, while the adaptive reward signals have to adjust the replay buffer size and the weight decay rate. Finally, we also notice that the proposed reward signals cannot effectively decrease U l * on Simple Topology. However, when we consider Moderate Topology and Complex Topology, the best reductions of U l * are bigger than 10%. The reason is that all rewards can approach the optimal policies on Simple Topology, which leaves no space for the proposed rewards to further improve. But when the topology becomes more complex, the proposed rewards begin to show their abilities. In short, our are: both global reward and local reward are prone to produce suboptimal policies; the mixed reward signals are off-the-shelf to learn better policies; and the adaptive reward signals can achieve best at the cost of careful experimental setting. Those also generalize the following thesis in single-agent RL to MARL: the training objective can differ from the performance objective, but still do well with respect to it. So we hope that MARL researchers can rethink the rewards used in their systems, especially when the agents cannot work as expected. In this section, we highlight some observations and considerations that inspire us to design the above reward signals, based on Simple Topology in Figure 1.The gR and dlR. We firstly try gR without any thinking, but only get a 30% convergence rate, so we try the widely used dlR, and we find the performance is slightly better. However, we notice that the maximum or minimum link utilization ratio of most divergence experiments is U f c or U f d, as shown in Figure 5. That is to say, the dlR cannot catch information of L f c or L f d, which is a little far from the nearest agent e (router f is not a ACCNet agent). We realise that the combination of gR and dlR might have the ability to catch both nearest link information and far away link information, so we propose the dlgM ixedR. Figure 5: link utilization ratios of one divergence environment using dlR.The dlgMixedR. As expected, the dlgM ixedR performs better than gR or dlR alone, not only for higher convergence rate but also lower U l *. At this time, we further ask ourselves two questions. The first question is that can we simplify dlgM ixedR but still keep its ability? This inspires us to propose the blR and subsequently blgM ixedR. The second one is that can we gradually decay the weight of dlR so that the received reward is progressively approaching gR, i.e., the performance objective of our environments? This inspires us to try the adaptive rewards. The blR and blgMixedR. Although blR is simpler than dlgM ixedR, it can incorporate both nearest link information and far away link information adaptively. And as expected, blR achieves similar convergence rates and maximum link utilization ratios as dlgM ixedR. But what surprises us is that blgM ixedR can boost the convergence rate up to 80%. One hindsight is that blR can better balance f actoredness and learnability. The dlgAdaptR and blgAdaptR. Although the adaptive rewards get the best , it is not until we actually implement them that we realize the difficulties. The are sensitive to replay buffer size and weight decay rate, because the reward in replay buffer is slightly inconsistent with current reward even thought the <s,a,s'> tuple is the same. Larger buffer size and higher weight decay rate usually mean greater inconsistency, while small buffer size and low weight decay rate in slow learning and data inefficiency. So we suggest to use the off-the-shelf mixed reward signals. Other reward forms. We also test min-max reward mmR = 1 + min(U l) − max(U l) and average reward aveR = 1−average(U l). The observation is that some links always have low link utilization ratios, which means that the agents have not learned to share flow on those links. So we try to encode those information into the reward using mmR and aveR. Although those reward forms take some effect indeed, we do not consider them as general methods to discuss as they are not what rewards. Finally, we give the link utilization ratios testing on real flow trajectory from the American Abilene Network, as shown in Figure 6. We see that all links have similar utilization ratios, and the trends of the curves are consistent. Those mean that all the links can properly share responsibility for the flow demand according to their respective capabilities. Figure 6: link utilization ratios testing on real Abilene Network flow using blgM ixedR. Why the convergence rate of Complex Topology is low? In this paper, we only focus on designing special reward signals, rather than applying other sophisticated technologies, to solve the packet routing problem. In fact, the convergence rate can be improved to almost 100% for all topologies if we combine the proposed rewards with other methods. To make this paper easy to read, we do not introduce irrelevant methods. Can the proposed rewards generalize successfully in other environments? In fact, not all environments can directly calculate the local reward or global reward, as BID24 point out. In such environments, the proposed rewards might be only useful at high computation cost. However, the calculation of the rewards is not the research purpose of this paper. We argue that although the proposed rewards have limitations, they can be easily applied to many real-world applications such as internet packet routing and traffic flow allocation, as mentioned in Section 3.Can the designed rewards be seen as a kind of auxiliary task? Yes, they are some auxiliary reward signals indeed. But please note that the auxiliary reward signals are different from the auxiliary task used in UNREAL BID15, where the auxiliary task is used for improving the feature extraction ability of neural networks, while our auxiliary reward signals directly guide the learned policies. In fact, the mixed rewards are similar with VIME as analyzed in Section 2.1. And the adaptive rewards are similar with curriculum learning BID38, as both of them train the agents progressively from easy to the final difficult environment. In this paper, we study reward design problem in cooperative MARL based on packet routing environments. Firstly, we show that both of the widely adopted global and local reward signals are prone to produce suboptimal policies. Then, inspired by some observations and considerations, we design some mixed reward signals, which are off-the-shelf to learn better policies. Finally, we turn the mixed reward signals into the adaptive counterparts, which achieve best in our experiments. Our study generalizes the following thesis BID6 BID35 in singleagent RL to MARL: the training objective that differs from the performance objective can still do well with respect to it. As reward design is a very fundamental problem in RL and especially in MARL, we hope that MARL researchers can rethink the rewards used in their systems. For future work, we would like to use Evolutionary Algorithm BID11 to search the best weight of local reward, and verify whether the learned weight has the same decay property. We also expect to test the proposed reward signals in other application domains. Real flow trajectories from the American Abilene Network are shown in FIG5. Note that we normalize the flow demands so that they can be consistent with link capacities. To test the learned policies, we randomly change the flow demands of each IE-pair. State is represented as tuple <F, U, A, aveU>, where F stands for two ticks history of flow demands; U stands for five ticks history of direct link utilization ratio; A stands for last action token by the agent; aveU stands for ten ticks average of direct link utilization ratio. Specifically, for Simple Topology in Figure 1, the state dimensions of agent a, b and e are 28, 28 and 41, respectively; for Moderate Topology in Figure 2, the state dimensions of agent a, b and e are 30, 30 and 41, respectively; for Complex Topology in Figure 3, the state dimensions of agent a, b, 1, 2, 3, 4, and 5 are 30, 30, 28, 30, 30, 30 and 30, respectively. For action, as the ingress-router should generate a splitting ratio y k i with a constraint k y k i = 1 for current traffic demand F i. So the softmax activation is chosen as the final layer of actor network. This design is natural for the continuous action with sum-to-one constraint. Settings related with ACCNet are: buffer size is 6280; batch size is 64; learning rates of actor and critic are 0.001 and 0.01, respectively; discount factor is 0.9; weight for updating target network is 0.001. The link utilization ratios of AC-CNet and A-CCNet testing on Complex Topology using Abilene Network flow and dlgM ixedR are shown in FIG6 and 9, respectively. A-CCNet achieves smaller fluctuation range of link utilization ratio than AC-CNet, which means that A-CCNet is better than AC-CNet to deal with this environment, as claimed in the original paper BID20. Besides, similar with Figure 6, all links in FIG6 and 9 have similar utilization ratios and the trends of the curves are consistent, which means that all the links can properly share responsibility for the flow demand according to their respective capabilities.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We study reward design problem in cooperative MARL based on packet routing environments. The experimental results remind us to be careful to design the rewards, as they are really important to guide the agent behavior.

Represent the short summary for finding the corresponding scientific article (long)

We use reinforcement learning for query reformulation on two tasks and surprisingly find that when training multiple agents diversity of the reformulations is more important than specialisation.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Recent advances have illustrated that it is often possible to learn to solve linear inverse problems in imaging using training data that can outperform more traditional regularized least squares solutions. Along these lines, we present some extensions of the Neumann network, a recently introduced end-to-end learned architecture inspired by a truncated Neumann series expansion of the solution map to a regularized least squares problem. Here we summarize the Neumann network approach, and show that it has a form compatible with the optimal reconstruction function for a given inverse problem. We also investigate an extension of the Neumann network that incorporates a more sample efficient patch-based regularization approach. We consider solving linear inverse problems in imaging in which a p-pixel image, β ∈ R p (in vectorized form), is observed via m noisy linear projections as y = Xβ +, where X ∈ R m×p and ∈ R m is a noise vector. The problem of estimating β from y is referred to as image reconstruction, and a typical estimate is given by solving the regualarized least squares problem β = arg min where r(·) is a regularizer. Classical image reconstruction methods specify a choice of regularizer to promote piecewise smoothness of the reconstruction, sparsity in some dictionary or basis, or other geometric properties. However, an emerging body of research explores the idea that training data can be used to learn to solve inverse problems using neural networks. At a high level, existing learning-based approaches to solving inverse problems can be categorized as either decoupled or end-to-end. Decoupled approaches first learn a representation of the data that is independent of the forward model X, followed by a reconstruction phase that uses X explicitly. Existing methods in this vein include using training images to learn a low-dimensional image manifold captured by the range of a generative adversarial network (GAN) and constraining the estimate β to lie on this manifold, or learning a denoising autoencoder that can be treated as a regularization step (i.e., proximal operator) within an iterative reconstruction scheme. End-to-end approaches incorporate the forward model X directly into the network architecture during both training and testing, and are optimized for a specific X or class of X's. Many end-to-end approaches are based on "unrolling" finitely many iterations of an optimization algorithm for solving, where instances of the regularizer (or its gradient or proximal operator) are replaced by a neural network to be trained; see among others. The advantage of a decoupled approach is that the learned representation can be used for a wide variety of inverse problems without having to retrain. However, this flexibility comes with a high price in terms of sample complexity. Learning a generative model or a denoising autoencoder fundamentally amounts to estimating a probability distribution and its support over the space of images; let us denote this distribution as P (β). On the other hand, if X is known at training time, then we only need to learn the conditional distribution P (β|Xβ), which can require far fewer samples to estimate. To make this idea more precise, consider the problem of finding the MSE optimal reconstruction function in the noiseless setting: Then ρ * is characterized as follows. Proposition 1. Let X ∈ R m×p, m ≤ p, be full rank, and let X ⊥ ∈ R p−m×p be a matrix whose rows form an orthonormal basis for the nullspace of X. Then the MSE-optimal reconstruction function ρ * in is given by where X + is the pseudoinverse of X and We omit the proof of Proposition 1 for brevity, but the technique is similar to those used in to derive the expressions for the MSE optimal autoencoder for a given data distribution. This proposition shows that the optimal reconstruction function only requires estimating a conditional expectation of the component of the image in the nullspace of the linear forward model, or implicitly a conditional probability density rather than the full probability density over the space of all images. Therefore, in settings where training data is limited, end-to-end approaches are expected to outperform decoupled approaches due to their lower sample complexity. It also implies the end-to-end networks should have a structure compatible with if they are to well-approximate the MSE optimal reconstruction function. The focus of this work is on the recently-proposed Neumann network architecture as an end-to-end approach for learning to solve inverse problems. Here we summarize the Neumann network architecture, and give it a new interpretation in light of Proposition 1. We also introduce an extension of Neumann networks to the case of a patch-based regularization strategy, which further improves the sample complexity of the approach. Neumann networks are motivated by the regularized least squares optimization problem in the special case where the regularizer r is quadratic. In particular, assume r(β) = 1 2 β Rβ so that ∇r(β) = Rβ for some matrix R ∈ R p×p. A necessary condition for β to be a minimizer of in this case is (X X + R)β = X y If the matrix on the left-hand side of is invertible, the solution is given by To approximate the matrix inverse in, the authors of use a Neumann series expansion for the inverse of a linear operator, given by A −1 = η ∞ k=0 (I − ηA) k, which converges provided I − ηA < 1. Applying this series expansion to the matrix inverse appearing in, we have β = ∞ j=0 (I − ηX X − ηR) j (ηX y). Truncating this series to B + 1 terms, and replacing multiplication by the matrix R with a general learnable mapping R: R p → R p, motivates an estimator β of the form The estimator above becomes trainable by letting R = R θ be a neural network depending on a vector of parameters θ ∈ R q to be learned from training data, along with the scale parameter η. Neumann Net Figure 1: Neumann network architecture. Unlike other networks based on unrolling of iterative optimization algorithms, the series structure of Neumann networks lead naturally to additional "skip connections" (highlighted in red) that route the output of each dashed block to directly to the output layer. Any estimator β(y) = β(y; θ, η) specified with trainable network R = R θ is called a Neumann network in. Figure 1 shows a block diagram which graphically illustrates a Neumann network. The main architectural difference with Neumann networks over related unrolling approaches is the presence of additional "skip connections" that arise naturally due to the series structure. Empirical evidence in suggests these additional skip connections may improve the optimization landscape relative to other architectures, and make Neumann networks easier to train. Efficiently finding a solution to the linear system using an iterative method can be challenging when the matrix X X + R is ill-conditioned. This suggests that the Neumann network, which is derived from a Neumann series expansion of the system in, may benefit from preconditioning. Starting from, for any λ > 0 we have (X X + λI)β + (R − λI)β = X y. Applying T λ:= (X X + λI) −1 to both sides and rearranging terms gives (I − λT λ +R)β = T λ X y. Following the same steps used to derive the Neumann network gives the modified estimator which is called a preconditioned Neumann network in. HereR =R θ is a trainable mapping depending on parameters θ. The Neumann network estimators in and can be interpreted as approximating the MSE optimal reconstruction function in Proposition 1. To see this, observe that the pseudo-inverse X + y = (X X) −1 X y is given by the Neumann series The preconditioned Neumann network estimator β(y) has the form where β R (y) collects all terms that depend on R. The preconditioned Neumann network more directly approximates ρ * (y) since the initial iterateβ = T λ X y = (X X + λI) −1 X y already well-approximates X + y provided λ > 0 is small. Here we present an extension to the Neumann network which incorporates a learned patchwise regularizer. For large images, learning an accurate regularizer may require more samples than are practical to gather due to cost or time constraints, leading to inaccurate reconstructions or overfitting. However, empirical evidence suggests there is considerable low-rank and other subspace structure shared among small patches of natural images. Redundancy and subspace structure across image patches permits learning parameters of statistical models for image patches using training data, like Gaussian mixture models with low-rank covariance structure. We propose leveraging the highly structured nature of image patches in the learned component of the Neumann network. Specifically, the patchwise learned regularizer first divides the input image into overlapping patches, subtracting the mean from each patch (a standard preprocessing technique in patch-based methods ), and passing each mean-subtracted patch through the learned component (e.g., neural network). The original patch means are added to the regularizer outputs, which are recombined. Figure 2 compares the presented learning-based methods at different training set sizes. Methods that do not incorporate the forward model, like ResAuto and CSGM, appear not to perform well in the low-sample regime. We also demonstrate that patchwise regularization enables reconstruction of large images with very small training sets. In this experiment, the training set consists only of a single clean image, taken from the SpaceNet dataset.Test PSNR is 31.90 ± 1.42 dB for the 8x8 patchwise regularized NN, and 18.34 ± 1.31 for the full-image regularized NN across a test set of size 64. Fig. 3 contains some sample reconstructions of an image from the test set. This work explores the Neumann network architecture to solve linear inverse problems, which can be interpreted as an approximation of the MSE optimal reconstruction according to our Proposition 1. The Neumann network architecture also permits a learned patchwise regularizer, which learns the low-dimensional conditional distributions over image patches instead of the whole image. The Neumann network is empirically competitive with other state-of-the-art methods for inverse problems in imaging, and we demonstrate the ability to learn to regularize from a single training pair.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

Neumann networks are an end-to-end, sample-efficient learning approach to solving linear inverse problems in imaging that are compatible with the MSE optimal approach and admit an extension to patch-based learning.

Represent the TLDR for finding the corresponding scientific article

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the scientific article to find a short TLDR

Question: End-to-end task-oriented dialogue is challenging since knowledge bases are usually large, dynamic and hard to incorporate into a learning framework. We propose the global-to-local memory pointer (GLMP) networks to address this issue. In our model, a global memory encoder and a local memory decoder are proposed to share external knowledge. The encoder encodes dialogue history, modifies global contextual representation, and generates a global memory pointer. The decoder first generates a sketch response with unfilled slots. Next, it passes the global memory pointer to filter the external knowledge for relevant information, then instantiates the slots via the local memory pointers. We empirically show that our model can improve copy accuracy and mitigate the common out-of-vocabulary problem. As a , GLMP is able to improve over the previous state-of-the-art models in both simulated bAbI Dialogue dataset and human-human Stanford Multi-domain Dialogue dataset on automatic and human evaluation. Task-oriented dialogue systems aim to achieve specific user goals such as restaurant reservation or navigation inquiry within a limited dialogue turns via natural language. Traditional pipeline solutions are composed of natural language understanding, dialogue management and natural language generation BID32 BID28, where each module is designed separately and expensively. In order to reduce human effort and scale up between domains, end-to-end dialogue systems, which input plain text and directly output system responses, have shown promising based on recurrent neural networks BID10 BID14 and memory networks BID24. These approaches have the advantages that the dialogue states are latent without hand-crafted labels and eliminate the needs to model the dependencies between modules and interpret knowledge bases (KB) manually. However, despite the improvement by modeling KB with memory network BID0, end-to-end systems usually suffer from effectively incorporating external KB into the system response generation. The main reason is that a large, dynamic KB is equal to a noisy input and hard to encode and decode, which makes the generation unstable. Different from chit-chat scenario, this problem is especially harmful in task-oriented one, since the information in KB is usually the expected entities in the response. For example, in TAB0 the driver will expect to get the correct address to the gas station other than a random place such as a hospital. Therefore, pointer networks BID26 or copy mechanism BID8 ) is crucial to successfully generate system responses because directly copying essential words from the input source to the output not only reduces the generation difficulty, but it is also more like a human behavior. For example, in TAB0, when human want to reply others the Valero's address, they will need to "copy" the information from the table to their response as well. Therefore, in the paper, we propose the global-to-local memory pointer (GLMP) networks, which is composed of a global memory encoder, a local memory decoder, and a shared external knowledge. Unlike existing approaches with copy ability BID9 BID8, which the only information passed to decoder is the encoder hidden states, our model shares the external knowledge and leverages the encoder and the external knowledge to learn a global memory pointer and global contextual representation. Global memory pointer modifies the external knowledge by softly filtering words that are not necessary for copying. Afterward, instead of generating system responses directly, the local memory decoder first uses a sketch RNN to obtain sketch responses without slot values but sketch tags, which can be considered as learning a latent dialogue management to generate dialogue action template. Then the decoder generates local memory pointers to copy words from external knowledge and instantiate sketch tags. We empirically show that GLMP can achieve superior performance using the combination of global and local memory pointers. In simulated out-of-vocabulary (OOV) tasks in the bAbI dialogue dataset BID0, GLMP achieves 92.0% per-response accuracy and surpasses existing end-to-end approaches by 7.5% in full dialogue. In the human-human dialogue dataset, GLMP is able to surpass the previous state of the art on both automatic and human evaluation, which further confirms the effectiveness of our double pointers usage. Our model 1 is composed of three parts: global memory encoder, external knowledge, and local memory decoder, as shown in FIG0 (a). The dialogue history X = (x 1, . . ., x n) and the KB information B = (b 1, . . ., b l) are the input, and the system response Y = (y 1, . . ., y m) is the expected output, where n, l, m are the corresponding lengths. First, the global memory encoder uses a context RNN to encode dialogue history and writes its hidden states into the external knowledge. Then the last hidden state is used to read the external knowledge and generate the global memory pointer at the same time. On the other hand, during the decoding stage, the local memory decoder first generates sketch responses by a sketch RNN. Then the global memory pointer and the sketch RNN hidden state are passed to the external knowledge as a filter and a query. The local memory pointer returns from the external knowledge can copy text from the external knowledge to replace the sketch tags and obtain the final system response. Our external knowledge contains the global contextual representation that is shared with the encoder and the decoder. To incorporate external knowledge into a learning framework, end-to-end memory networks (MN) are used to store word-level information for both structural KB (KB memory) and temporal-dependent dialogue history (dialogue memory), as shown in FIG0. In addition, the MN is well-known for its multiple hop reasoning ability BID24, which is appealing to strengthen copy mechanism. Global contextual representation. In the KB memory module, each element b i ∈ B is represented in the triplet format as (Subject, Relation, Object) structure, which is a common format used to represent KB nodes BID18. For example, the KB in the TAB0 will be denoted as {(Tom's house, distance, 3 miles),..., (Starbucks, address, 792 Bedoin St)}. On the other hand, the dialogue context X is stored in the dialogue memory module, where the speaker and temporal encoding are included as in BID0 like a triplet format. For instance, the first utterance from the driver in the TAB0 will be denoted as {($user, turn1, I), ($user, turn1, need), ($user, turn1, gas)}. For the two memory modules, a bag-of-word representation is used as the memory embeddings. During the inference time, we copy the object word once a memory position is pointed to, for example, 3 miles will be copied if the triplet (Toms house, distance, 3 miles) is selected. We denote Object function as getting the object word from a triplet. Knowledge read and write. Our external knowledge is composed of a set of trainable embedding matrices C = (C 1, . . ., C K+1), where C k ∈ R |V |×d emb, K is the maximum memory hop in the MN, |V | is the vocabulary size and d emb is the embedding dimension. We denote memory in the external knowledge as M = [B; X] = (m 1, . . ., m n+l), where m i is one of the triplet components mentioned. To read the memory, the external knowledge needs a initial query vector q 1. Moreover, it can loop over K hops and computes the attention weights at each hop k using DISPLAYFORM0 where DISPLAYFORM1 is the embedding in i th memory position using the embedding matrix C k, q k is the query vector for hop k, and B is the bag-of-word function. Note that p k ∈ R n+l is a soft memory attention that decides the memory relevance with respect to the query vector. Then, the model reads out the memory o k by the weighted sum over c k+1 and update the query vector q k+1. Formally, DISPLAYFORM2 In FIG1 (a), a context RNN is used to model the sequential dependency and encode the context X. Then the hidden states are written into the external knowledge as shown in FIG0 (b). Afterward, the last encoder hidden state serves as the query to read the external knowledge and get two outputs, the global memory pointer and the memory readout. Intuitively, since it is hard for MN architectures to model the dependencies between memories, which is a serious drawback especially in conversational related tasks, writing the hidden states to the external knowledge can provide sequential and contextualized information. With meaningful representation, our pointers can correctly copy out words from external knowledge, and the common OOV challenge can be mitigated. In addition, using the encoded dialogue context as a query can encourage our external knowledge to read out memory information related to the hidden dialogue states or user intention. Moreover, the global memory pointer that learns a global memory distribution is passed to the decoder along with the encoded dialogue history and KB information. Context RNN. A bi-directional gated recurrent unit (GRU) BID2 ) is used to encode dialogue history into the hidden states H = (h e 1, . . ., h e 1), and the last hidden state h e n is used to query the external knowledge as the encoded dialogue history. In addition, the hidden states H are written into the dialogue memory module in the external knowledge by summing up the original memory representation with the corresponding hidden states. In formula, c DISPLAYFORM0 Global memory pointer. Global memory pointer G = (g 1, . . ., g n+l) is a vector containing real values between 0 and 1. Unlike conventional attention mechanism that all the weights sum to one, each element in G is an independent probability. We first query the external knowledge using h e n until the last hop, and instead of applying the Softmax function as in FORMULA0, we perform an inner product followed by the Sigmoid function. The memory distribution we obtained is the global memory pointer G, which is passed to the decoder. To further strengthen the global pointing ability, we add an auxiliary loss to train the global memory pointer as a multi-label classification task. We show in the ablation study that adding this additional supervision does improve the performance. Lastly, the memory readout q K+1 is used as the encoded KB information. In the auxiliary task, we define the label DISPLAYFORM1 by checking whether the object words in the memory exists in the expected system response Y. Then the global memory pointer is trained using binary cross-entropy loss Loss g between G and G label. In formula, DISPLAYFORM2 Given the encoded dialogue history h e n, the encoded KB information q K+1, and the global memory pointer G, our local memory decoder first initializes its sketch RNN using the concatenation of h e n and q K+1, and generates a sketch response that excludes slot values but includes the sketch tags. For example, sketch RNN will generate "@poi is @distance away", instead of "Starbucks is 1 mile away." At each decoding time step, the hidden state of the sketch RNN is used for two purposes: 1) predict the next token in vocabulary, which is the same as standard sequence-to-sequence (S2S) learning; 2) serve as the vector to query the external knowledge. If a sketch tag is generated, the global memory pointer is passed to the external knowledge, and the expected output word will be picked up from the local memory pointer. Otherwise, the output word is the word that generated by the sketch RNN. For example in FIG1 (b), a @poi tag is generated at the first time step, therefore, the word Starbucks is picked up from the local memory pointer as the system output word. DISPLAYFORM0 We use the standard cross-entropy loss to train the sketch RNN, we define Loss v as. DISPLAYFORM1 We replace the slot values in Y into sketch tags based on the provided entity table. The sketch tags ST are all the possible slot types that start with a special token, for example, @address stands for all the addresses and @distance stands for all the distance information. Local memory pointer. Local memory pointer L = (L 1, . . ., L m) contains a sequence of pointers. At each time step t, the global memory pointer G first modify the global contextual representation using its attention weights, DISPLAYFORM2 and then the sketch RNN hidden state h d t queries the external knowledge. The memory attention in the last hop is the corresponding local memory pointer L t, which is represented as the memory distribution at time step t. To train the local memory pointer, a supervision on top of the last hop memory attention in the external knowledge is added. We first define the position label of local memory pointer L label at the decoding time step t as DISPLAYFORM3 The position n+l+1 is a null token in the memory that allows us to calculate loss function even if y t does not exist in the external knowledge. Then, the loss between L and L label is defined as DISPLAYFORM4 Furthermore, a record R ∈ R n+l is utilized to prevent from copying same entities multiple times. All the elements in R are initialized as 1 in the beginning. During the decoding stage, if a memory position has been pointed to, its corresponding position in R will be masked out. During the inference time,ŷ t is defined aŝ DISPLAYFORM5 where is the element-wise multiplication. Lastly, all the parameters are jointly trained by minimizing the weighted-sum of three losses (α, β, γ are hyper-parameters): DISPLAYFORM6 3 EXPERIMENTS We use two public multi-turn task-oriented dialogue datasets to evaluate our model: the bAbI dialogue BID0 and Stanford multi-domain dialogue (SMD). The bAbI dialogue includes five simulated tasks in the restaurant domain. Task 1 to 4 are about calling API calls, modifying API calls, recommending options, and providing additional information, respectively. Task 5 is the union of tasks 1-4. There are two test sets for each task: one follows the same distribution as the training set and the other has OOV entity values. On the other hand, SMD is a human-human, multi-domain dialogue dataset. It has three distinct domains: calendar scheduling, weather information retrieval, and point-of-interest navigation. The key difference between these two datasets is, the former has longer dialogue turns but the regular user and system behaviors, the latter has few conversational turns but variant responses, and the KB information is much more complicated. The model is trained end-to-end using Adam optimizer BID12, and learning rate annealing starts from 1e −3 to 1e −4. The number of hop K is set to 1,3,6 to compare the performance difference. The weights α, β, γ summing up the three losses are set to 1. All the embeddings are initialized randomly, and a simple greedy strategy is used without beam-search during the decoding stage. The hyper-parameters such as hidden size and dropout rate are tuned with grid-search over Table 2: Per-response accuracy and completion rate (in the parentheses) on bAbI dialogues. GLMP achieves the least out-of-vocabulary performance drop. Baselines are reported from Query Reduction Network BID20, End-to-end Memory Network BID0, Gated Memory Network BID15, Point to Unknown Word BID9, and Memory-to-Sequence. DISPLAYFORM0 Ptr-Unk Mem2Seq GLMP K1 GLMP K3 GLMP K6 T1 99.4 (-) 99.9 (99.6) 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 T2 99.5 (-) 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 100 FORMULA0 the development set (per-response accuracy for bAbI Dialogue and BLEU score for the SMD). In addition, to increase model generalization and simulate OOV setting, we randomly mask a small number of input source tokens into an unknown token. The model is implemented in PyTorch and the hyper-parameters used for each task and the dataset statistics are reported in the Appendix. bAbI Dialogue. In Table 2, we follow BID0 to compare the performance based on per-response accuracy and task-completion rate. Note that for utterance retrieval methods, such as QRN, MN, and GMN, cannot correctly recommend options (T3) and provide additional information (T4), and a poor generalization ability is observed in OOV setting, which has around 30% performance difference in Task 5. Although previous generation-based approaches (Ptr-Unk, Mem2Seq) have mitigated the gap by incorporating copy mechanism, the simplest cases such as generating and modifying API calls (T1, T2) still face a 6-17% OOV performance drop. On the other hand, GLMP achieves a highest 92.0% task-completion rate in full dialogue task and surpasses other baselines by a big margin especially in the OOV setting. No per-response accuracy loss for T1, T2, T4 using only the single hop, and only decreases 7-9% in task 5.Stanford Multi-domain Dialogue. For human-human dialogue scenario, we follow previous dialogue works BID10 to evaluate our system on two automatic evaluation metrics, BLEU and entity F1 score 2. As shown in TAB3, GLMP achieves a highest 14.79 BLEU and 59.97% entity F1 score, which is a slight improvement in BLEU but a huge gain in entity F1. In fact, for unsupervised evaluation metrics in task-oriented dialogues, we argue that the entity F1 might be a more comprehensive evaluation metric than per-response accuracy or BLEU, as shown in that humans are able to choose the right entities but have very diversified responses. Note that the of rule-based and KVR are not directly comparable because they simplified the task by mapping the expression of entities to a canonical form using named entity recognition and linking 3. Moreover, human evaluation of the generated responses is reported. We compare our work with previous state-of-the-art model Mem2Seq 4 and the original dataset responses as well. We randomly select 200 different dialogue scenarios from the test set to evaluate three different responses. Amazon Mechanical Turk is used to evaluate system appropriateness and human-likeness on a scale from 1 to 5. As the shown in TAB3, we see that GLMP outperforms Mem2Seq in both measures, which is coherent to previous observation. We also see that human performance on this assessment sets the upper bound on scores, as expected. More details about the human evaluation are reported in the Appendix. Ablation Study. The contributions of the global memory pointer G and the memory writing of dialogue history H are shown in TAB4. We compare the using GLMP with K = 1 in bAbI OOV setting and SMD. GLMP without H means that the context RNN in the global memory encoder does not write the hidden states into the external knowledge. As one can observe, our model without H has 5.3% more loss in the full dialogue task. On the other hand, GLMP without G means that we do not use the global memory pointer to modify the external knowledge, and an 11.47% entity F1 drop can be observed in SMD dataset. Note that a 0.4% increase can be observed in task 5, it suggests that the use of global memory pointer may impose too strong prior entity probability. Even if we only report one experiment in the table, this OOV generalization problem can be mitigated by increasing the dropout ratio during training. Visualization and Qualitative Evaluation. Analyzing the attention weights has been frequently used to interpret deep learning models. In Figure 3, we show the attention vector in the last hop for each generation time step. Y-axis is the external knowledge that we can copy, including the KB information and the dialogue history. Based on the question "what is the address?" asked by the driver in the last turn, the gold answer and our generated response are on the top, and the global memory pointer G is shown in the left column. One can observe that in the right column, the final memory pointer successfully copy the entity chevron in step 0 and its address 783 Arcadia Pl in step 3 to fill in the sketch utterance. On the other hand, the memory attention without global weighting is reported in the middle column. One can find that even if the attention weights focus on several point of interests and addresses in step 0 and step 3, the global memory pointer can mitigate the issue as expected. More dialogue visualization and generated including several negative examples and error analysis are reported in the Appendix. Task-oriented dialogue systems. Machine learning based dialogue systems are mainly explored by following two different approaches: modularized and end-to-end. For the modularized systems BID29 BID28, a set of modules for natural language understanding BID32 BID1, dialogue state tracking BID13 BID35, dialogue management BID23, and natural language generation BID22 are used. These approaches achieve good stability via combining domain-specific knowledge and slot-filling techniques, but additional human labels are needed. On the other hand, end-to-end approaches have shown promising recently. Some works view the task as a next utterance retrieval problem, for examples, recurrent entity networks share parameters between RNN BID30, query reduction networks modify query between layers BID20, and memory networks BID0 BID15 perform multi-hop design to strengthen reasoning ability. In addition, some approaches treat the task as a sequence generation problem. BID14 Delexicalized Generation: @poi is at @address Final Generation: chevron is at 783_arcadia_pl Gold: 783_arcadia_pl is the address for chevron gas_station Figure 3: Memory attention visualization in the SMD navigation domain. Left column is the global memory pointer G, middle column is the memory pointer without global weighting, and the right column is the final memory pointer. these approaches can encourage more flexible and diverse system responses by generating utterances token-by-token. Pointer network. BID26 uses attention as a pointer to select a member of the input source as the output. Such copy mechanisms have also been used in other natural language processing tasks, such as question answering BID3 BID10, neural machine translation BID9 BID8, language modeling BID17, and text summarization BID19. In task-oriented dialogue tasks, first demonstrated the potential of the copy-augmented Seq2Seq model, which shows that generationbased methods with simple copy strategy can surpass retrieval-based ones. Later, augmented the vocabulary distribution by concatenating KB attention, which at the same time increases the output dimension. Recently, combines end-to-end memory network into sequence generation, which shows that the multi-hop mechanism in MN can be utilized to improve copy attention. These models outperform utterance retrieval methods by copying relevant entities from the KBs. Others. BID10 proposes entity indexing and introduces recorded delexicalization to simplify the problem by record entity tables manually. In addition, our approach utilized recurrent structures to query external memory can be viewed as the memory controller in Memory augmented neural networks (MANN) BID6. Similarly, memory encoders have been used in neural machine translation BID27 and meta-learning applications. However, different from other models that use a single matrix representation for reading and writing, GLMP leverages end-to-end memory networks to perform multiple hop attention, which is similar to the stacking self-attention strategy in the Transformer BID25. In the work, we present an end-to-end trainable model called global-to-local memory pointer networks for task-oriented dialogues. The global memory encoder and the local memory decoder are designed to incorporate the shared external knowledge into the learning framework. We empirically show that the global and the local memory pointer are able to effectively produce system responses even in the out-of-vocabulary scenario, and visualize how global memory pointer helps as well. As a , our model achieves state-of-the-art in both the simulated and the human-human dialogue datasets, and holds potential for extending to other tasks such as question answering and text summarization. A.1 TRAINING PARAMETERS Table 5: Selected hyper-parameters in each dataset for different hops. The values is the embedding dimension and the GRU hidden size, and the values between parenthesis is the dropout rate. For all the models we used learning rate equal to 0.001, with a decay rate of 0.5. T2 T3 T4 T5 SMD GLMP K1 64 FORMULA0 2) GLMP K3 64 FORMULA0 2) GLMP K6 64 FORMULA0 A.2 DATASET STATISTICS For bAbI dialogues, the mistakes are mainly from task 3, which is recommending restaurants based on their rating from high to low. We found that sometimes the system will keep sending those restaurants with the higher score even if the user rejected them in the previous turns. On the other hand, SMD is more challenging for response generation. First, we found that the model makes mistakes when the KB has several options corresponding to the user intention. For example, once the user has more than one doctor appointment in the table, the model can barely recognize. In addition, since we do not include the domain specific and user intention supervision, wrong delexicalized responses may be generated, which in an incorrect entity copy. Lastly, we found that the copied entities may not be matched to the generated sketch tags. For example, an address tag may in a distance entity copy. We leave the space of improvement to future works. One of the reviewers suggested us to compare our work to some existing dialogue framework such as PyDial 5. To the best of our knowledge, in the PyDial framework, it requires to have the dialogue acts labels for the NLU module and the belief states labels for the belief tracker module. The biggest challenge is we do not have such labels in the SMD and bAbI datasets. Moreover, the semi tracker in PyDial is rule-based, which need to re-write rules whenever it encounters a new domain or new datasets. Even its dialogue management module could be a learning solution like policy networks, the input of the policy network is still the hand-crafted state features and labels. Therefore, without the rules and labels predefined in the NLU and belief tracker modules, PyDial could not learn a good policy network. Truly speaking, based on the data we have (not very big size) and the current state-of-the-art machine learning algorithms and models, we believe that a well and carefully constructed task-oriented dialogue system using PyDial in a known domain using human rules (in NLU and Belief Tracker) with policy networks may outperform the end-to-end systems (more robust). However, in this paper, without additional human labels and human rules, we want to explore the potential and the advantage of end-to-end systems. Besides easy to train, for multi-domain cases, or even zero-shot domain cases, we believe end-to-end approaches will have better adaptability compared to any rule-based systems. Delexicalized Generation: the nearest @poi_type is @poi, @distance away at @address Final Generation: the nearest grocery_store is willows_market, 3_miles away at 409_bollard_st Gold: we are 3_miles away from willows_market but there is a car_collision_nearby Figure 5: Memory attention visualization from the SMD navigation domain. Delexicalized Generation: the nearest @poi_type is @poi, @distance away at @address Final Generation: the nearest grocery_store is sigona_farmers_market, 4_miles away at 819_alma_st Gold: there are whole_foods 2_miles away and sigona_farmers_market 4_miles away where do we go? Delexicalized Generation: okay, setting a reminder for your @event with @party @date at @time Final Generation: okay, setting a reminder for your dinner with marie 7pm at the_6th Gold: i will set a reminder for dinner at 7pm, for the_6th of this month with marie Figure 9: Memory attention visualization from the SMD schedule domain. Delexicalized Generation: your @event is on @date at @time Final Generation: your tennis_activity is on the_4th at 5pm Gold: your tennis_activity is on the_4th at 5pm and your sister will be attending Delexicalized Generation: your next @event is on @date at @time Final Generation: your next tennis_activity is on the_13th at 6pm Gold: your next tennis_activity is on the_4th at 5pm with your mother Delexicalized Generation: your @event is on @date at @time with @party Final Generation: your doctor is on the_5th at 6pm with alex Gold: are you talking about the doctor_appointment on wednesday or the one on the_5th? Delexicalized Generation: it will not be @weather_attribute in @location @weekly_time Final Generation: it will not be drizzle in redwood_city weekend Gold: there will be no drizzle in redwood_city this weekend Delexicalized Generation: there will be @weather_attribute in @location on @date Final Generation: there will be clear_skies in danville on thursday Gold: dew is predicted in danville on thursday FIG0: Memory attention visualization from the SMD weather domain.

GLMP: Global memory encoder (context RNN, global pointer) and local memory decoder (sketch RNN, local pointer) that share external knowledge (MemNN) are proposed to strengthen response generation in task-oriented dialogue.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Combining graph neural networks and the RNN graph generative model, we propose a novel architecture that is able to learn from a sequence of evolving graphs and predict the graph topology evolution for the future timesteps


Question: represent the article for finding a single-sentence tldr

Question: The checkerboard phenomenon is one of the well-known visual artifacts in the computer vision field. The origins and solutions of checkerboard artifacts in the pixel space have been studied for a long time, but their effects on the gradient space have rarely been investigated. In this paper, we revisit the checkerboard artifacts in the gradient space which turn out to be the weak point of a network architecture. We explore image-agnostic property of gradient checkerboard artifacts and propose a simple yet effective defense method by utilizing the artifacts. We introduce our defense module, dubbed Artificial Checkerboard Enhancer (ACE), which induces adversarial attacks on designated pixels. This enables the model to deflect attacks by shifting only a single pixel in the image with a remarkable defense rate. We provide extensive experiments to support the effectiveness of our work for various attack scenarios using state-of-the-art attack methods. Furthermore, we show that ACE is even applicable to large-scale datasets including ImageNet dataset and can be easily transferred to various pretrained networks. The checkerboard phenomenon is one of the well-known artifacts that arise in various applications such as image super resolution, generation and segmentation BID29 BID27. In general, the checkerboard artifacts indicate an uneven pattern on the output of deep neural networks (DNNs) that occurs during the feed-forward step. BID21 have investigated in-depth the origin of the phenomenon that the artifacts come from the uneven overlap of the deconvolution operations (i.e., transposed convolution,) on pixels. Its solutions have been suggested in various studies BID35 BID0.Interestingly, a possible relationship between the checkerboard artifacts and the robustness of neural networks has been noted by BID21 but not has it been seriously investigated. Moreover, while the previous works BID16 BID20 BID21 BID8 have concentrated on the artifacts in the pixel space, studies have been rare on the artifacts in the gradient space that occur during the backward pass of the convolution operation. To show that the gradient checkerboard artifacts phenomenon is crucial for investigating the network robustness and is indeed a weak point of a neural network, we focus on analyzing its effects on the gradient space in terms of adversarial attack and defense. By explicitly visualizing the gradients, we demonstrate that the phenomenon is inherent in many contemporary network architectures such as ResNet BID12, which use strided convolutions with uneven overlap. It turns out that the gradient checkerboard artifacts substantially influence the shape of the loss surface, and the effect is image-agnostic. Based on the analysis, we propose an Artificial Checkerboard Enhancer module, dubbed ACE. This module further boosts or creates the checkerboard artifacts in the target network and manipulates the gradients to be caged in the designated area. Because ACE guides the attacks to the intended environment, the defender can easily dodge the attack by shifting a single pixel (Figure 1) with a negligible accuracy loss. Moreover, we demonstrate that our module is scalable to large-scale datasets such as ImageNet BID6 ) and also transferable to other models in a plug and play fashion without additional fine-tuning of pretrained networks. Therefore, our module is highly practical in general scenarios in that we can easily plug any pretrained ACE module into the target architecture. Base Network (e.g. ResNet, VGG) ACE Module The first layer of the ACE module has kernel size 1 and stride 2 Gradient Checkerboard Artifact (GCA) Generate adversarial image from the model's enhanced gradient checkerboard artifact Adv Image GCA Induce attack on GCA Zero-pad a single row/column Remove opposite row/column Adv Image Input ImageFigure 1: Defense procedure using the proposed Artificial Checkerboard Enhancer (ACE) module. ACE shapes the gradient into a checkerboard pattern, thus attracting adversarial attacks to the checkerboard artifacts. Since the defender is aware of the guided location in advance, adversarial attacks can be easily deflected during inference by padding the image with a single row/column and discarding the opposite row/column. Our contributions are summarized as three-fold:Analysis of gradient checkerboard artifacts. We investigate the gradient checkerboard artifacts in depth which are inherent in many of the contemporary network architectures with the uneven overlap of convolutions. To the best of our knowledge, this is the first attempt to analyze the artifacts of the gradient space in terms of network robustness. We empirically show that the gradient checkerboard artifacts incur vulnerability to the network. Artificial Checkerboard Enhancer (ACE). We introduce ACE module that strengthens gradient artifacts and induces adversarial attacks to our intended spaces. After guiding the attacks to the pre-specified area using ACE, we deflect adversarial attacks by one-pixel padding. Our extensive experimental support that our proposed defense mechanism using ACE module successfully defends various adversarial attacks on CIFAR-10 and ImageNet BID6 datasets. Scalability. We show that ACE is readily transferable to any pretrained model without fine-tuning, which makes the module scalable to a large-scale dataset. To the best of our knowledge, this is the first defense method that attempts and succeeds to defend the attacks with the projected gradient descent algorithm (PGD) BID17 BID1 on ImageNet dataset. Adversarial attacks can be conducted in various ways depending on how much the adversary, also known as the threat model, has access to the target model. If the attacker can acquire the gradients of the target, gradient-based attacks can be performed, which are usually very effective because iterative optimization becomes possible BID10 BID14; BID22 BID5. Score-based attacks can be valid when the adversary can use the logits or the predicted probabilities for generating adversarial examples BID32. If generated adversarial examples are not from the target model, we call this transfer-based attack. Recently, a new type of attack, called a decisionbased attack, has been introduced where the adversary only has knowledge about the final decision of the model (e.g., top-1 class label) BID3.According to, defense methods can be largely categorized into gradient masking and adversarial training. Gradient masking methods usually make adversaries difficult to compute exact gradients and make it challenging to fool the target. Gradient obfuscation, which is a recently introduced term of gradient masking by BID2, includes specific gradient categories such as stochastic gradients, shattered gradients and vanishing/exploding gradients. Works related to our defense method can be considered as input transformations which focus on the input image BID36 BID25 BID4 BID37 BID11.On the other hand, adversarial training has been known to make models robust against adversarial perturbation BID17 BID19 BID33, but there remains an issue that the robustness comes with the cost of accuracy BID34 BID31. Moreover, according to BID28, restricting on l ∞ bounded perturbations during adversarial training has limited robustness to attacks with different distortion metrics. We would like to define the following terms here and will use them without further explanation throughout this paper. Gradient Overlap (Ω(x i)) represents the number of parameters associated with a single pixel in the input x. For more explanation on its calculation, see Appendix C. We define the set of pixels whose gradient overlap is in the top p fraction as G(p) (e.g., G(1.0) represents the entire pixel set). Gradient Checkerboard Artifacts (GCA) is a phenomenon which shows checkerboard patterns in gradients. The existence of GCA has been introduced in BID21, although not has it been thoroughly examined. GCA occurs when a model uses a convolution operation of kernel size k that is not divisible by its stride s. We first introduce a simple experiment that motivated the design of our proposed ACE module. We conduct an experiment to visualize the attack success rate of a given image set using a single pixel perturbation attack. For each pixel of an input image, we perturb the pixel to white (i.e., a pixel with RGB value of ). Next, we use our toy model based on LeNet (see Appendix A for the detailed architecture) and ResNet-18 BID12 to measure the attack success rate on the test images in CIFAR-10 dataset. Note that the attack success rate P attack in Figure 2 is defined as the average number of successful attacks per pixel on the entire set of test images. As we can see in Figure 2a and Figure 2c, checkerboard patterns in the attack success rate are clearly observable. This pattern can be considered as image-agnostic because it is the of the average over the entire set of the test images of CIFAR-10 dataset. Then a natural question arises: What is the cause of this image-agnostic phenomenon? We speculate that the uneven gradient overlap is the cause of this phenomenon, which is directly associated with the number of parameters that are connected to a single pixel. As depicted in Figure 2b and Figure 2d, we can observe checkerboard patterns in the gradient overlap. In fact, this uneven overlap turns out to be substantially susceptible to the adversarial attacks. We will provide the supporting on this in the following sections.(a) Pattack of Toy model DISPLAYFORM0 Figure 2: Illustration of the attack success rate P attack and the gradient overlap Ω(x i) of toy model and ResNet-18. The illustrated gradient overlap of ResNet-18 comes from the features after the fifth convolutional layer. Attack success rate (a) and (c) are computed by perturbing each pixel of an image to white (i.e., ), over the entire set of test images in CIFAR-10 dataset. Note that higher probability at a pixel denotes a higher success rate when it is attacked. We can observe patterns in P attack aligned to our gradient overlap on (b) toy model and (d) ResNet-18. Table 1: Top-1 test accuracy (%) after performing various adversarial attacks on every pixel G(p = 1.0), its subset G(p = 0.3), and their differences (i.e., diff) on CIFAR-10 dataset. The toy model (see Appendix A) and ResNet-18 achieved 81.4% and 94.6% top-1 test accuracy, respectively. Note that all the diffs are close to zero. ResNet-18 Attack Methods DISPLAYFORM0 OnePixel BID32 56.6 58.4 1.7 57.2 59.5 2.4 JSMA BID22 0.2 0.4 0.2 3.2 9.8 6.6 DeepFool BID18 18.5 18.6 0.1 7.2 11.5 4.3 CW BID5 0.0 0.0 0.0 0.0 0.0 0.0 PGD BID17 0.0 1.6 1.6 0.0 0.0 0.0 To show that the pixels with high gradient overlaps are indeed a weak point of network, we generate adversarial examples on G(p). We evaluate top-1 accuracy of our toy model (the model defined as in the previous subsection) and ResNet-18 on CIFAR-10 dataset after performing five adversarial attacks BID32 BID22 BID18 BID5 BID17 for p ∈ {1.0, 0.3} (Table 1). Interestingly, constraining the domain of the attacks to G(0.3) barely decreases the success rate compared to the attacks on G(1.0).We can observe that the pixels with the high gradient overlaps are more susceptible (i.e., likely to be in a vulnerable domain) to the adversarial attacks. Considering all the observations, we leverage the vulnerable domain of the pixels for adversarial defense. If we can intentionally impose the GCA onto a model input and let GCA occupy the vulnerable domain, we can fully induce the attacks on it so that the induced attacks can be dodged easily by a single padding operation. In this section, we propose the Artificial Checkerboard Enhancer (ACE) module, which artificially enhances the checkerboard pattern in the input gradients so that it induces the vulnerable domain to have the identical pattern. Figure 3a illustrates our proposed ACE module, which is based on a convolutional autoencoder. The encoder consists of convolutional layers where the first layer's k is not divisible by s (k ≡ 0 mod s), for example, when k = 1 and s = 2. In order to preserve the information of the input x, we add an identity skip connection that bypasses the input of ACE module to the output. The hyperparameter λ is to control the magnitude of checkerboard artifacts in the input gradients. We plug our ACE module in front of a base convolutional network to enhance the checkerboard artifacts on its input gradients. By increasing λ, we can artificially increase the gradient checkerboard artifacts of the network. Figure 3b and 3c show the heatmaps of the input gradients of ResNet-18 when λ = 10 and λ = 0 (i.e., without ACE module), respectively. The heatmap is generated by a channel-wise absolute sum of input gradients. Note that the checkerboard pattern is clearly observed in Figure 3b.By changing the value of λ, we report the top-1 test accuracy and the proportion of the pixels having checkerboard artifacts (C) in the top-30% gradient overlaps G(0.3) (TAB0). More precisely, we denote C as the GCA imposed by ACE module, which is identical to the set of pixels that are connected to its first convolutional layer of k = 1 and s = 2. In TAB0, we can observe that 1) there is only a small decrease in the accuracy even with a large λ and 2) the pixels with the large gradient overlaps gradually coincide with the GCA as the λ increases. Furthermore, according to the in Table 1, the existing adversarial attacks tend to be induced on the pixels with the high gradient overlaps. Therefore, we can conjecture that our ACE module which builds a high gradient overlap with a significant checkerboard pattern could cage the adversarial attacks into the checkerboard artifacts, and this will be empirically proven in Section 5. We now study the effects of λ. To this end, we first visualize the classified labels with respect to the magnitude of the perturbation on pixels in checkerboard artifacts C and pixels in non-checkerboard artifacts X\C. Let x be the input image andê C = DISPLAYFORM0 where M C denotes a mask (i.e., value of i-th element of M C equals to one if x i ∈ C and zero otherwise) and denotes the element-wise multiplication. We defineê X\C in a similar way. We plot the classified label map of x +īê X\C +jê C by varyingī andj. For the experiment, we first train ACE module as an autoencoder using ImageNet BID6 ) datasets and plug ACE module into pretrained ResNet-152 as described in the following experiment section. Next, we plot classified labels for a sample image from ImageNet by varyingī andj from −100 to 100 by interval 1 (i.e., we test 40k perturbed images per each image) in FIG1. The figure signifies the classified label map with respect to the perturbation on X\C and C. Without using our ACE module (when λ = 0), the perturbation through non-artifact pixels and artifact pixels similarly affect the label map. However, when λ > 0, we can observe that the artifact pixels are susceptible to change their labels with only a small perturbation while non-artifact pixels are robust to the same perturbation. Note that the asymmetry between the artifact pixels and non-artifacts becomes more clear as λ increases. Here, we propose a novel defense method using ACE module. First, we plug ACE module into the input of a given network which enhances gradient checkerboard artifacts. Next, we let the adversary generate adversarial images using the network with ACE module. Because ACE module is likely to expose the pixels in the vulnerable domain, which is empirically shown in Appendix I, we may consider the ACE module as an inducer to the adversarial perturbations generated by the adversary into the checkerboard. Interestingly, because the pixels in the vulnerable domain are going to be aligned to the repeated checkerboard pattern, by shifting a single pixel of the adversarial sample, we can move perturbations into the non-vulnerable domain (i.e., non-artifact pixels). The proposed defense mechanism is similar to the defense method introduced in BID36. However, thanks to our ACE module, the vulnerable pixels are induced to checkerboard artifacts so that only onepixel padding is enough to avoid several adversarial attacks aiming the pixels in the vulnerable domain. We also report the defense regarding the diverse padding-sizes in Appendix D. It is worthwhile to note that large λ induces significant gradient checkerboard artifacts hence leads to more robust defense. The detailed are reported in Section 5. For thorough evaluation, we evaluate our proposed defense method in the following three attack scenarios, which are vanilla, transfer, and adaptive attacks. First, in the vanilla attack scenario, the adversary has access to the target model but not our proposed defense method (i.e., single pixel padding defense). We remark that the vanilla attack scenario is similar to the scenario used by BID36. Second, in the transfer attack scenario, the adversary generates adversarial perturbations from a source model, which is different from the target model. Finally, in the adaptive attack scenario, the adversary knows every aspect of the model and the defense method so that it can directly exploit our defense. For our experiments, Expectation Over Transformation (EOT) BID2 ) is used for the adaptive attack scenario. For the evaluation of our defense method, we use the following five attack methods as our adversary. OnePixel BID32, JSMA BID22, DeepFool , CW BID5 and PGD BID17 1. In addition, we conduct experiments on CIFAR-10 and ImageNet BID6 ) datasets for the attack scenarios. For the models evaluated in CIFAR-10 dataset, we train the models with the two layered ACE module from scratch. Note that the first convolutional layer in ACE has k = 1 and s = 2, and the following deconvolutional layer has k = 3 and s = 2 so that it enhances gradient checkerboard artifacts. We would like to recall that the top-1 accuracy of VGG-11 BID30 and ResNet-18 BID12 with ACE module in respect of different λ are reported in TAB0. Meanwhile, for a large-scale dataset such as ImageNet, training an entire network is very expensive. Hence, we train the ACE module as autoencoder with UNet architecture BID26 and plug ACE into the input of a pretrained network without any additional training procedure. In order to retain the scale of the input image of the pretrained network, we slightly modify the ACE module by constraining λ ∈ and multiplying (1 − λ) to the identity skip connection. In this way, our ACE module becomes capable of handling large-scale datasets in a plug and play fashion on any model. We now introduce two evaluation metrics named attack survival rate and defense success rate which denote that top-1 accuracy after attack divided by the original top-1 accuracy of the model Table 3: Attack survival rate (%) and defense success rate (%) (larger is better) by one-pixel padding defense on CIFAR-10 dataset with varying λ. Note that λ = 0 is the equivalent setting to single padding pixel experiments in BID36 Table 4: Attack survival rate (%) and defense success rate (%) (larger is better) by one-pixel padding defense on ImageNet dataset with varying λ. ResNet-18-ACE 0.0 0.0 10.2 23.9 32.1 93.5 98.3 VGG-11-ACE 0.8 1.7 1.9 9.7 85.9 79.5 98.8 and top-1 accuracy after defending attack divided by the original accuracy, respectively. We note that all experimental reported in this section are reproduced by ourselves 2.Vanilla attack scenario. We evaluate our defense method in both CIFAR-10 and ImageNet dataset. For CIFAR-10 experiments, we train an entire network including ACE. For ImageNet experiments, we only train ACE module as conventional autoencoder by minimizing the mean squared error without training an entire network. Table 3 shows that our proposed method defends various attack methods successfully on CIFAR-10 dataset. We remark that by choosing a large λ (e.g., λ = 100), the top-1 accuracy after defense on performed attacks is barely dropped. In table 4, we report same experiments conducted on ImageNet dataset. We use the pretrained models of VGG-19 and ResNet-152 3 repository whose top-1 test accuracy on ImageNet dataset is 72.38% and 78.31%, respectively. We abbreviate the of OnePixel, JSMA and DeepFool due to the infeasibly high cost of time and memory limit for those algorithms to craft numerous adversarial images in ImageNet dataset. Comparison with other defense methods BID25 BID36 BID4 for CIFAR-10 and ImageNet datasets are reported in Appendix E. To investigate the effectiveness of λ regards to defense success rate, we evaluate PGD attack for λ ∈ {0, 1, 2, 5, 10, 20, 100} and report the defense success rates in Table 5. The shows that: Defense success rate (%) on CIFAR-10 test dataset after one-pixel padding defense on transfer attacks from the source model to the target model via JSMA, CW and PGD. The number followed by name of network denotes the intensity of λ, e.g., VGG-11-100 denotes VGG-11 + ACE with λ = 100. Note that adversarial examples generated by different λ are not transferable to other models.when λ increases, the defense success rate improves as well. To the best of our knowledge, this is the first work that defends PGD up to 98%.Transfer attack scenario. It has been demonstrated that conventional deep networks are vulnerable to transfer attacks proposed by BID23. To show that our method is robust to transfer attacks, we conduct transfer attacks for VGG-11 and ResNet-18 by choosing λ ∈ {0, 10, 100} on CIFAR-10 dataset. We report the defense success rate after one-pixel padding defense of transfer attacks by JSMA, CW and PGD in FIG2. More including OnePixel and DeepFool transfer attack experiments are reported in Appendix G. The show that generated adversarial samples are not transferable to other models with different λ. Adaptive attack scenario. A successful defense method should defend l 0, l 2, and even l ∞ bounded adversaries and also show robust on the adaptive white-box setting. We report our defense combined with adversarial training against Expectation Over Transformation (EOT) BID2 ) of PGD attack in Appendix H. From the , it turns out that our method is complementary with robust training defense methods (e.g., adversarial training). Therefore, if we combine our method with the existing robust training defense methods together, we can secure promising on the vanilla scenario and even perform well on the adaptive scenario. In this paper, we have investigated the gradient checkerboard artifacts (GCA) which turned out to be a potential threat to the network robustness. Based on our observations, we proposed a novel Artificial Checkerboard Enhancer (ACE) which attracts the attack to the pre-specified domain. ACE is a module that can be plugged in front of any pretrained model with a negligible performance drop. Exploiting its favorable characteristics, we provide a simple yet effective defense method that is even scalable to a large-scale dataset such as ImageNet. Our extensive experiments show that the proposed method can deflect various attack scenarios with remarkable defense rates compared with several existing adversarial defense methods. We present our toy model used in Section 3. The model consists of three convolutional layers followed by ReLUs and two fully-connected layers (Table 6). Table 6: Architecture detail of our toy model used on CIFAR-10 dataset. Input image x ∈ R 32×32×3 3×3, stride=2 conv16 ReLU 3×3, stride=2 conv32 ReLU 3×3, stride=1 conv64 ReLU dense 1600 → 512 dense 512 → 10 To visualize the existence of checkerboard artifacts in the gradients, here we present the average heatmaps over the test images. Let ∇x l i,j,k denote the feature gradient in layer l where i, j and k are the indices of xy-axis and the channel, respectively. In each layer, the heatmap h l is gathered by a channel-wise absolute sum of the feature gradients. More specifically, h We can observe checkerboard patterns before strided convolutional layers (layer 5, 9, 13) and they propagate to their front layers (layer 4, 8). To count the gradient overlaps per pixel (Ω(x i)), for simplicity, we only consider convolutional and fully-connected layers. We set each and every network parameter to 1 to evenly distribute the gradients from the loss layer to the previous layers. Therefore, the gradients of the modified network are aggregated in a certain location (or pixel) if and only if there is a linear overlap from the later layers at the backward pass. In TAB2, we report the defense success rate varying the padding-size from one to five. We can observe that the proposed defense mechanism almost prevents accuracy drop with the padding-sizes of one, three and five (odd numbers). The supports that ACE induces adversarial perturbations into checkerboard artifacts, which could be avoided by padding an image with one pixel. BID25 BID36 BID4 ) by following their papers. For fair comparison, we follow the suggested settings in their papers, and the are presented in TAB3. Specifically, for BID25, with R-CAM implemented, the number of deflection is set to 100 with window-size of 10 and sigma for denoiser of 0.04, respectively. For BID36, the scale factor is set to 0.9. Finally, for BID4, the number of the ensemble is set to 1000 and the radius of the region is set to 0.02. In this section, we plot the classified label map after pixel perturbation on artifact pixels and nonartifact pixels as described in Section 4.2. We follow the same setting of Section 4.2. The are reported in FIG5. Adaptive attack case A successful defense method should be able to defend various conditions including l 0, l 2 and l ∞ -bounded adversaries as well as an adaptive white-box setting where the adversary knows our defense method in every aspect. Under the adaptive white-box setting, we conducted experiments in Table 10. In order to avoid direct exploitation of our padding direction, we shift our images in the random direction around the known safe points near our checkerboard artifacts. By combining PGD adversarial training BID17 for robustness on l ∞ bounded attacks to our method, we can defend the corresponding adaptive attack for stochastic methods known as Expectation Over Transformation (EOT, BID1). This method was used to break BID36 in BID2. Although we have some loss in Top-1 accuracy when λ is high, we have advantages in that we can defend vanilla attack cases at the same time. Table 10: Top-1 accuracy (%) after EOT attack on our defense method together with adversarial training on CIFAR-10 dataset. All images were padded by one-pixel to X-axis. Conducted attacks are written in the format of PGD-norm-iterations. ACE module only shows training accuracy loss due to high λ.

We propose a novel aritificial checkerboard enhancer (ACE) module which guides attacks to a pre-specified pixel space and successfully defends it with a simple padding operation.

Represent the TLDR for finding the corresponding scientific article.

Batch normalization reduces adversarial robustness, as well as general robustness in many cases, particularly to noise corruptions.