Question: Represent the scientific article to find a short TLDR.

Question: Training neural networks to be certifiably robust is critical to ensure their safety against adversarial attacks. However, it is currently very difficult to train a neural network that is both accurate and certifiably robust. In this work we take a step towards addressing this challenge. We prove that for every continuous function $f$, there exists a network $n$ such that: (i) $n$ approximates $f$ arbitrarily close, and (ii) simple interval bound propagation of a region $B$ through $n$ yields a that is arbitrarily close to the optimal output of $f$ on $B$. Our can be seen as a Universal Approximation Theorem for interval-certified ReLU networks. To the best of our knowledge, this is the first work to prove the existence of accurate, interval-certified networks. Much recent work has shown that neural networks can be fooled into misclassifying adversarial examples , inputs which are imperceptibly different from those that the neural network classifies correctly. Initial work on defending against adversarial examples revolved around training networks to be empirically robust, usually by including adversarial examples found with various attacks into the training dataset (; ; ; ; ; ;). However, while empirical robustness can be practically useful, it does not provide safety guarantees. As a , much recent research has focused on verifying that a network is certifiably robust, typically by employing methods based on mixed integer linear programming , SMT solvers , semidefinite programming (a), duality (; b), and linear relaxations (; ; b; ; ;). Because the certification rates were far from satisfactory, specific training methods were recently developed which produce networks that are certifiably robust:; Raghunathan et al. (2018b); Wang et al. (2018a);;; train the network with standard optimization applied to an over-approximation of the network behavior on a given input region (the region is created around the concrete input point). These techniques aim to discover specific weights which facilitate verification. There is a tradeoff between the degree of the over-approximation used and the speed of training and certification. Recently, (b) proposed a statistical approach to certification, which unlike the non-probabilistic methods discussed above, creates a probabilistic classifier that comes with probabilistic guarantees. So far, some of the best non-probabilistic achieved on the popular MNIST and CIFAR10 datasets have been obtained with the simple Interval relaxation , which scales well at both training and verification time. Despite this progress, there are still substantial gaps between known standard accuracy, experimental robustness, and certified robustness. For example, for CIFAR10, the best reported certified robustness is 32.04% with an accuracy of 49.49% when using a fairly modest l ∞ region with radius 8/255 . The state-of-the-art non-robust accuracy for this dataset is > 95% with experimental robustness > 50%. Given the size of this gap, a key question then is: can certified training ever succeed or is there a fundamental limit? In this paper we take a step in answering this question by proving a parallel to the Universal Approximation Theorem . We prove that for any continuous function f defined on a compact domain Γ ⊆ R m and for any desired level of accuracy δ, there exists a ReLU neural network n which can certifiably approximate f up to δ using interval bound propagation. As an interval is a fairly imprecise relaxation, our directly applies to more precise convex relaxations (e.g., ;). Theorem 1.1 (Universal Interval-Certified Approximation, Figure 1). Let Γ ⊂ R m be a compact set and let f: Γ → R be a continuous function. For all δ > 0, there exists a ReLU network n such that for all boxes [a, b] in Γ defined by points a, b ∈ Γ where a k ≤ b k for all k, the propagation of the box [a, b] using interval analysis through the network n, denoted n ([a, b]), approximates the set We recover the classical universal approximation theorem (|f (x) − n(x)| ≤ δ for all x ∈ Γ) by considering boxes [a, b] describing points (x = a = b). Note that here the lower bound is not [l, u] as the network n is an approximation of f. Because interval analysis propagates boxes, the theorem naturally handles l ∞ norm bound perturbations to the input. Other l p norms can be handled by covering the l p ball with boxes. The theorem can be extended easily to functions f: Γ → R k by applying the theorem component wise. Practical meaning of theorem The practical meaning of this theorem is as follows: if we train a neural network n on a given training data set (e.g., CIFAR10) and we are satisfied with the properties of n (e.g., high accuracy), then because n is a continuous function, the theorem tells us that there exists a network n which is as accurate as n and as certifiable with interval analysis as n is with a complete verifier. This means that if we fail to find such an n, then either n did not possess the required capacity or the optimizer was unsuccessful. Focus on the existence of a network We note that we do not provide a method for training a certified ReLU network -even though our method is constructive, we aim to answer an existential question and thus we focus on proving that a given network exists. Interesting future work items would be to study the requirements on the size of this network and the inherent hardness of finding it with standard optimization methods. Universal approximation is insufficient We now discuss why classical universal approximation is insufficient for establishing our . While classical universal approximation theorems state that neural networks can approximate a large class of functions f, unlike our , they do not state that robustness of the approximation n of f is actually certified with a scalable proof method (e.g., interval bound propagation). If one uses a non scalable complete verifier instead, then the standard Universal approximation theorem is sufficient. To demonstrate this point, consider the function f: R → R (Figure 2b) mapping all x ≤ 0 to 1, all x ≥ 1 to 0 and all 0 < x < 1 to 1 − x and two ReLU networks n 1 (Figure 2a) and n 2 (Figure 2c) perfectly approximating f, that is n 1 (x) = f (x) = n 2 (x) for all x. For δ = 1 4, the interval certification that n 1 maps all However, interval certification succeeds for n 2, because n 2 =. To the best of our knowledge, this is the first work to prove the existence of accurate, interval-certified networks. After adversarial examples were discovered by , many attacks and defenses were introduced (for a survey, see). Initial work on verifying neural network robustness used exact methods on small networks, while later research introduced methods based on over-approximation (; a; ;) aiming to scale to larger networks. A fundamentally different approach is randomized smoothing (; Lécuyer et al., 2019; b), in which probabilistic classification and certification with high confidence is performed. As neural networks that are experimentally robust need not be certifiably robust, there has been significant recent research on training certifiably robust neural networks (b; ; 2019; ; ; a; ; a; ; b). As these methods appear to have reached a performance wall, several works have started investigating the fundamental barriers in the datasets and methods that preclude the learning of a robust network (let alone a certifiably robust one) (; ;). In our work, we focus on the question of whether neural networks are capable of approximating functions whose robustness can be established with the efficient interval relaxation. Feasibility Results with Neural Networks Early versions of the Universal Approximation Theorem were stated by and. showed that networks using sigmoidal activations could approximate continuous functions in the unit hypercube, while showed that even networks with only one hidden layer are capable of approximating Borel measurable functions. (2019a) provide an explicit construction to obtain the network. We note that both of these works focus on Lipschitz continuous functions, a more restricted class than continuous functions, which we consider in our work. In this section we provide the concepts necessary to describe our main . Adversarial Examples and Robustness Verification Let n: R m → R k be a neural network, which classifies an input x to a label t if n(x) t > n(x) j for all j = t. For a correctly classified input x, an adversarial example is an input y such that x is imperceptible from y to a human, but is classified to a different label by n. Frequently, two images are assumed to be "imperceptible" if there l p distance is at most. The l p ball around an image is said to be the adversarial ball, and a network is said to be -robust around x if (Figure 3a) using a ReLU network n = ξ 0 + k n k. The ReLU networks n k (Figure 3c) approximate the N -slicing of f (Figure 3b), as a sum of local bumps (Figure 6). every point in the adversarial ball around x classifies the same. In this paper, we limit our discussion to l ∞ adversarial balls which can be used to cover to all l p balls. The goal of robustness verification is to show that for a neural network n, input point x and label t, every possible input in an l ∞ ball of size around x (written B ∞ (x)) is also classified to t., and λ ∈ R ≥0. Furthermore, we used to distinguish the function f from its interval-transformation f. To illustrate the difference between f and f, consider f (illustrating the loss in precision that interval analysis suffers from. Interval analysis provides a sound over-approximation in the sense that for all function f, the values that Furthermore all combinations f of +, −, · and R are monotone, that is for . This will later be needed. In this section, we provide an explanation of the proof of our main , Theorem 4.6, and illustrate the main points of the proof. The first step in the construction is to deconstruct the function f into slices {f k : Γ → [0, for all x, where ξ 0 is the minimum of f (Γ). We approximate each slice f k by a ReLU network δ 2 · n k. The network n approximating f up to δ will be n(x):= ξ 0 + δ 2 k n k (x). The construction relies on 2 key insights, (i) the output of δ 2 · n k can be confined to the interval [0, δ 2], thus the loss of analysis precision is at most the height of the slice, and (ii) we can construct the networks n k using local bump functions, such that only 4 slices can contribute to the loss of analysis precision, two for the lower interval bound, two for the upper one. The slicing {f k} 0≤k<5 of the function f: [−2, 2] → R (Figure 3a), mapping x to f (x) = −x 3 + 3x is depicted in Figure 3b. The networks n k are depicted in Figure 3c. In this example, evaluating the interval-transformer of n, namely n on the box Definition 4.1 (N -slicing (Figure 3b) ). Let Γ ⊂ R m be a closed m-dimensional box and let f: Γ → R be continuous. The N -slicing of f is a set of functions {f k} 0≤k<N defined by where To construct a ReLU network satisfying the desired approximation property (Equation) if evaluated on boxes in B(Γ), we need the ReLU network nmin capturing the behavior of min as a building block (similar to). It is given by With the ReLU network nmin, we can construct recursively a ReLU network nmin N mapping N arguments to the smallest one (Definition A.8). Even though the interval-transformation loses precision, we can establish bounds on the precision loss of nmin N sufficient for our use case (Appendix A). Now, we use the clipping function R [*,1]:= 1 − R(1 − x) clipping every value exceeding 1 back to 1 (Figure 5) to construct the local bumps φ c w.r.t. a grid G. G specifies the set of all possible local bumps we can use to construct the networks n k. Increasing the finesse of G will increases the approximation precision. Definition 4.2 (local bump, Figure 6). M } ⊆ G be a set of grid points describing the corner points of a hyperrectangle in G. We define a ReLU neural network φ c: We will describe later how M and c get picked. A graphical illustration of a local bump for in two dimensions and c = {Figure 6 . The local bump φ c (x) evaluates to 1 for all x that lie within the convex hull of c, namely conv(c), after which φ c (x) quickly decreases linearly to 0. φ c has 1 + 2(2d − 1) + 2d ReLUs and 1 + log 2 (2d + 1) + 1 layers. The formal proof is given in Appendix A. The next lemma shows, how a ReLU network n k can approximate the slice f k while simultaneously confining the loss of analysis precision. Lemma 4.4. Let Γ ⊂ R m be a closed box and let f: Γ → R be continuous. For all δ > 0 there exists a set of ReLU networks {n k} 0≤k<N of size N ∈ N approximating the N -slicing of f, {f k} 0≤k<N (ξ k as in Definition 4.1) such that for all boxes B ∈ B(Γ) and It is important to note that in Equation we mean f and not f. The proof for Lemma 4.4 is given in Appendix A. In the following, we discuss a proof sketch. Because Γ is compact and f is continuous, f is uniformly continuous by the Heine-Cantor Theorem. So we can pick a M ∈ N such that for all x, y ∈ Γ satisfying ||y−x|| Next, we construct for every slice k a set ∆ k of hyperrectangles on the grid G: if a box B ∈ B(Γ) fulfills f (B) ≥ ξ k+1 + δ 2, then we add a minimal enclosing hyperrectangle c ⊂ G such that B ⊆ conv(c) to ∆ k, where conv(c) denotes the convex hull of c. This implies, using uniform continuity of f and that the grid G is fine enough, that f (conv(c)) ≥ ξ k+1. Since there is only a finite number of possible hyperrectangles in G, the set ∆ k is clearly finite. The network fulfilling Equation is where φ c is as in Definition 4.2. The n k are depicted in Figure 3c. Now, we see that Equation holds by construction: For all boxes B ∈ B(Γ) such that f ≥ ξ k+1 + δ 2 on B exists c ∈ ∆ k such that B ⊆ conv(c) which implies, using Lemma 4.3, that φ c (B) =, hence Similarly, if f (B) ≤ ξ k − δ 2 holds, then it holds for all c ∈ ∆ k that B does not intersect N (conv(c)). Indeed, if a c ∈ ∆ k would violate this, then by construction, f (conv(c)) ≥ ξ k+1, contradicting f (B) ≤ ξ k − where l:= min f (B) and u:= max f (B). Proof. Pick N such that the height of each slice is exactly δ 2, if this is impossible choose a slightly smaller δ. Let {n k} 0≤k<N be a series of networks as in Lemma 4.4. Recall that ξ 0 = min f (Γ). We define the ReLU network Let B ∈ B(Γ). Thus we have for all k Let p, q ∈ {0, . . ., N − 1} such that Figure 7: Illustration of the proof for Theorem 4.5. as depicted in Figure 7. Thus by Equation for all k ∈ {0, . . ., p − 2} it holds that n k (B) = and similarly, by Equation for all k ∈ {q + 2, . . ., N − 1} it holds that n k (B) =. Plugging this into Equation after splitting the sum into three parts leaves us with Applying the standard rules for interval analysis, leads to where we used in the last step, that ξ 0 + k δ 2 = ξ k. For all terms in the sum except the terms corresponding to the 3 highest and lowest k we get Indeed, from Equation we know that there is Similarly, from Equation we know, that there is x ∈ B such that f (x) ≥ ξ q = ξ q−1 + We know further, that if p + 3 ≤ q, than there is an x ∈ B such that f (x) ≥ ξ p+3 = ξ p+2 + If p + 3 > q the lower bound we want to prove becomes vacuous and only the upper one needs to be proven. Thus we have where l:= min f (B) and u:= max f (B). where l, u ∈ R m such that l k:= min f (B) k and u k:= max f (B) k for all k. Proof. This is a direct consequence of using Theorem 4.5 and the Tietze extension theorem to produce a neural network for each dimension d of the codomain of f. Note that Theorem 1.1 is a special case of Theorem 4.6 with d = 1 to simplify presentation. We proved that for all real valued continuous functions f on compact sets, there exists a ReLU network n approximating f arbitrarily well with the interval abstraction. This means that for arbitrary input sets, analysis using the interval relaxation yields an over-approximation arbitrarily close to the smallest interval containing all possible outputs. Our theorem affirmatively answers the open question, whether the Universal Approximation Theorem generalizes to Interval analysis. Our address the question of whether the interval abstraction is expressive enough to analyse networks approximating interesting functions f. This is of practical importance because interval analysis is the most scalable non-trivial analysis. Lemma A.1 (Monotonicity). The operations +, − are monotone, that is for all Further the operation * and R are monotone, that is for all Proof. [Definition A.2 (N -slicing). Let Γ ⊂ R m be a compact m-dimensional box and let f: Γ → R be continuous. The N -slicing of f is a set of functions {f k} 0≤k≤N −1 defined by where Proof. Pick x ∈ Γ and let l ∈ {0, . . ., N − 1} such that ξ l ≤ f (x) ≤ ξ l+1. Then Definition A.4 (clipping). Let a, b ∈ R, a < b. We define the clipping function R [*,b]: R → R by Lemma A.5 (clipping). The function R [*,b] sends all x ≤ b to x, and all x > b to b. Further, Proof. We show the proof for R [a,b], the proof for R [*,b] is similar. Definition A.6 (nmin). We define the ReLU network nmin: Lemma A.7 (nmin). Let x, y ∈ R, then nmin(x, y) = min(x, y). Proof. Because nmin is symmetric in its arguments, we assume w.o.l.g. x ≥ y. Definition A.8 (nmin N). For all N ∈ N ≥1, we define a ReLU network nmin N defined by Proof. The symmetry on abstract elements is immediate. In the following, we omit some of to improve readability. Claim: R(a+c)−R(−a−c) = a+c. If a+c > 0 then −a−c < 0 thus the claim in this case. Indeed: So the expression simplifies to We proceed by case distinction: By symmetry of nmin equivalent to Case 1. Hence Case 3: a − d < 0 and b − c > 0: Proof. By induction. Base case: Induction hypothesis: The property holds for N s.t. 0 < N ≤ N − 1. Induction step: Then it also holds for N: Proof. By induction: Let N = 2: Lemma A.14 Induction hypothesis: The statement holds for all 2 ≤ N ≤ N − 1. Proof. Let N ∈ N such that N ≥ 2 ξmax−ξmin δ where ξ min:= min f (Γ) and ξ max:= max f (Γ). For simplicity we assume Γ = m. Using the Heine-Cantor theorem, we get that f is uniformly continuous, thus there exists a δ > 0 such that ∀x, y ∈ Γ.||y − x|| ∞ < δ ⇒ ||f (y) − f (x)|| <

represent the short summary for finding the corresponding scientific article (long)

We prove that for a large class of functions f there exists an interval certified robust network approximating f up to arbitrary precision.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, we propose an efficient framework to accelerate convolutional neural networks. We utilize two types of acceleration methods: pruning and hints. Pruning can reduce model size by removing channels of layers. Hints can improve the performance of student model by transferring knowledge from teacher model. We demonstrate that pruning and hints are complementary to each other. On one hand, hints can benefit pruning by maintaining similar feature representations. On the other hand, the model pruned from teacher networks is a good initialization for student model, which increases the transferability between two networks. Our approach performs pruning stage and hints stage iteratively to further improve the performance. Furthermore, we propose an algorithm to reconstruct the parameters of hints layer and make the pruned model more suitable for hints. Experiments were conducted on various tasks including classification and pose estimation. Results on CIFAR-10, ImageNet and COCO demonstrate the generalization and superiority of our framework. In recent years, convolutional neural networks (CNN) have been applied in many computer vision tasks, e.g. classification BID21; BID6, objects detection BID8; BID30, and pose estimation BID25. The success of CNN drives the development of computer vision. However, restricted by large model size as well as computation complexity, many CNN models are difficult to be put into practical use directly. To solve the problem, more and more researches have focused on accelerating models without degradation of performance. Pruning and knowledge distillation are two of mainstream methods in model acceleration. The goal of pruning is to remove less important parameters while maintaining similar performance of the original model. Despite pruning methods' superiority, we notice that for many pruning methods with the increase of pruned channel number, the performance of pruned model drops rapidlly. Knowledge distillation describes teacher-student framework: use high-level representations from teacher model to supervise student model. Hints method BID31 shares a similar idea of knowledge distillation, where the feature map of teacher model is used as high-level representations. According to BID36, the student network can achieve better performance in knowledge transfer if its initialization can produce similar features as the teacher model. Inspired by this work, we propose that pruned model outputs similar features with original model's and provide a good initialization for student model, which does help distillation. And on the other hand, hints can help reconstruct parameters and alleviate degradation of performance caused by pruning operation. FIG0 illustrates the motivation of our framework. Based on this analysis, we propose an algorithm: we do pruning and hints operation iteratively. And for each iteration, we conduct a reconstructing step between pruning and hints operations. And we demonstrate that this reconstructing operation can provide a better initialization for student model and promote hints step (See FIG1 . We name our method as PWH Framework. To our best knowledge, we are the first to combine pruning and hints together as a framework. Our framework can be applied on different vision tasks. Experiments on CIFAR- 10 , and Hints can help pruned model reconstruct parameters. And the network pruned from the teacher model can provide a good initialization for student model in hints learning.effectiveness of our framework. Furthermore, our method is a framework where different pruning and hints methods can be included. To summarize, the contributions of this paper are as follows: FORMULA0 We analyze the properties of pruning and hints methods and show that these two model acceleration methods are complementary to each other. To our best knowledge, this is the first work that combines pruning and hints. Our framework is easy to be extended to different pruning and hints methods. Sufficient experiments show the effectiveness of our framework on different datasets for different tasks. Recently, model acceleration has received a great deal of attention. Quantization methods BID29 BID5; BID20 BID39 reduce model size by quantizing float parameters to fixed-point parameters. And fixed-point networks can be speeded up on special implementation. Group convolution based methods BID16 BID3 separates a convolution operation into several groups, which can reduce computation complexity. Several works exploit linear structure of parameters and approach parameters using lowrank way to reduce computational parameters BID7 BID19; BID0. In our experiments, we use two of current mainstream model acceleration way: pruning and knowledge distillation. Network pruning has been proposed for a long time, such as BID11; BID12; BID22. Recent pruning methods can be roughly adopted in two levels, i.e. channel-wise BID28; BID14 BID32; BID17 and parameter-wise BID10; BID35; BID23 BID34; BID27.In this paper, we use channel-wise approach as our pruning method. There are many methods in channel-wise family. He et al. BID14 prune channels in LASSO regression way from sample feature map. Proposed in BID26, the scale parameters in Batch Normalization layers are used to evaluate the importance of different channels. BID28 use taylor formula to analyze each channel's contribution to the loss and prune the lowest contribution channel. We utilize this method in our framework. Despite the superiority of pruning methods, we find that the effectiveness of them will observably decrease with the increase of pruned channel numbers. Knowledge distillation (KD) BID15 is the pioneering work of this field. Hinton et al. BID15 define soft targets and use it to supervise student networks. Beyond soft targets, hints are introduced in Fitnets BID31, which can be explained as whole feature map mimic learning. Several researches have focused on hints. BID37 propose atloss to mimic combined output of an ensemble of large networks using student networks. Furthermore, Li et al. BID24 demonstrate a mimic learning strategy based on region of interests to improve small networks' performance for object detection. However, most of these works train student model from scratch and ignore the significance of student networks' initialization. In this section, we will describe our method in details. First we show hints and pruning methods which have been used in our framework. Then we introduce reconstructing operation and analyze its effectiveness. Finally, combining hints, pruning and reconstructing operation, we propose PWH Framework. The pruning method we use in this paper is based on BID28. The algorithm can be described as a combinatorial optimization problem: DISPLAYFORM0 Where C(·) is the cost function of the task, D is the training samples, W and W are the parameters of original and pruning networks. In the optimization problem, ||W || 0 bounds the number of nonzero parameters in W. The parameter W i whether to be pruned completely depends on its outputs h i. And the problem can be redescribed as minimizing ∆C(DISPLAYFORM1 However, it's hard to find global optimal solution of the problem. Using taylor expansion can get approximate formula of the objective function: DISPLAYFORM2 During backpropagation, we can get gradient δC δhi and activation h i . So this criteria can be easily implemented for channel pruning. Hints can provide an extra supervision for student network, and it usually combines with task loss. The whole loss function of hints learning is represented as follows: DISPLAYFORM0 Where L t is the task loss (e.g. classification loss) and L h is the hints loss. λ is hints loss weight which determines the intensity of hints supervision. There are many types of hints methods. Different hints methods are suitable for different tasks and network architectures. To demonstrate the superiority and generalization of our framework, we try three kinds of hints methods in our experiments: L2 hints, normalization hints and adaption hints. We introduce L2 hints, normalization hints in appendix. First we slim the original network with reducing certain number of channels. Then we reconstruct the hints layer of the pruned model to minimize the difference of feature map between teacher and student. Finally, we start hints step to advance the performance of pruned model. Adaption Hints demonstrates that it's necessary to add an adaption layer between student and teacher networks. The adaption layer can help transfer student layer feature space to teacher layer feature space, which will promote hints learning. The expression of adaption hints is described in equation 4. DISPLAYFORM1 Where r(·) is adaption layer. And for convolutional neural networks, adaption layer is 1 × 1 convolution layer. The objective function of reconstructing step is: DISPLAYFORM0 Where Y is the feature map of original (teacher) model, X is the input of hints layer and W is the parameter of hints layer. The optimization problem has close-form solution using least square method: DISPLAYFORM1 However, because some datasets (e.g. ImageNet BID6) have numerous images, X will be high-dimension matrix. And it's impossible to solve the problem which involves such huge matrix in one time. So we randomly sample images in dataset and compute X according to these images. Due to the randomness, the reconstructed weights may be worse than original weights. Thus, we finally use a switch to select better weights (See equation 6).W f = arg min DISPLAYFORM2 Where W f, W o and W r are the final weights, original weights and reconstructed weights of hints layer. Y 0 and X 0 are computed from the whole dataset. The objective function of Normalized L2 loss (See equation 16) is different from L2 loss, but we explain that the reconstruction step is also effective for normalized L2 loss. The details of proof is available in supplementary material. Combining pruning step, reconstructing step and hints step, we propose our PWH Framework. The framework iteratively conducts three steps. For pruning, we reduce the model size by certain num-bers of channels. Then the parameters of hints layer will be reconstructed to minimize the difference of feature map between pruned model's and teacher model's. Finally we use pruned model as student, original model as teacher and conduct hints learning. And in the next iteration, the original model will be substituted by the student model in this iteration. After training, the student model becomes the teacher model for the next iteration. And another hints step is implemented at the ending of the framework where the teacher model will be set as the original model at the beginning of training (the teacher model in the first iteration). The pseudocode of our approach is provided in appendix. We demonstrate that compared with the original model in the first iteration, the student model in the current iteration is a better candidate for the teacher model in next hints step. The reason is that the model before pruned and after pruned have more similar feature map and parameters, which can promote and speed up hints step. At the end of the whole framework, we do another hints step. Different from preceding hints step, the teacher model is selected as the original model in the first iteration. We demonstrate that the final hints step is like the finetune step in pruning methods, which may need long-time training and improves the performance of compressed model. And original model in the first iteration will be the better teacher. FIG1 shows the pipeline of our framework. The hints and pruning in PWH Framework are complementary to each other. On one hand, pruned model is a good initialization to student model in hints step. We propose that the feature map of pruned model is similar to original model's compared with random initialization model's. In this way, proposed in BID36, the transferability between student and teacher network will increase, which is beneficial for hints learning. Experiments in §4.4 demonstrate that the difference between original model's and pruned model's feature map is much smaller than random initialized model's. On the other hand, hints helps pruning reconstruct parameters. We demonstrate that when pruned channels number is large, pruning method is inefficient. The pruning operation will bring large degradation of performance in this case. We find that pruning numerous channels will destroy the main structure of networks(See 3). And hints can alleviate this trend and help recover the structure and reconstruct parameters in model(See 4).The motivation of reconstruct step is the generalization of our method. Our approach is a framework and it should be available for different pruning methods. However, the criteria of some pruning methods are not based on minimizing the reconstructing error of feature map. In other words, there is still room to improve the similarity between feature map of original (teacher) and pruned (student) networks, which is beneficial for hints learning. We only conduct reconstructing operation on hints layer because it can not only reduce the difference of feature map used for hints but also maintain the main structure of the pruned model (See experiments in 4.3.3). Moreover, for adaption hints methods, it need to initialize adaption layer(hint layer). Compared with random initialization, reconstruction operation can help to construct this layer and provide more similar features with teacher models'. We conduct experiments on CIFAR-10 , ImageNet BID6 and for classfication and pose estimation tasks to demonstrate the superiority of PWH Framework. In this section, we first introduce implementation details in different experiments on different datasets. Then we compare our method with pruning methods and hint methods. We train networks using PyTorch deep learning framework. Pruning-only refers to a classical iterative pruning and finetuning operation. And for hints-only methods, we set original model as teacher model and use the compressed random initialized model as student model. For fair comparison, the student model in hints-only shares the same network structure with student model in PHW Frame- TAB1 illustrates . We can find that PWH Framework outperforms pruning-only method and hints-only method for a large margin on all datasets for different tasks, which verify the effectiveness of our framework and also shows that hints and pruning can be combined to improve the performance. Results on different tasks and models show that PWH Framework can be implemented without the restriction of tasks and network architectures. Moreover, illustrated in table 1, our framework can be applied for different pruning ratios, which means that we can adjust pruning ratio in the framework for different tasks to achieve different acceleration ratios. To further analyze PWH Framework, we do several ablation studies. All experimetns are conducted on CIFAR-10 dataset using VGG16BN. The feature map proposed in this section refers to the output of last convolution layer, which is also the feature map used for hints learning. And in this section, we do ablation study for three different aspects. First, we do experiments to show iterative operation is an important component of PWH Framework. Then we study on the selection of teacher model in hints step. Finally, we validate on the effects of reconstructing step. In PWH Framework, we implement three steps iteratively. And in this section we will show the importance and necessity of iterative operation. We conduct an experiment to compare the effects The relationship between the performance of network and the number of pruned channels using different methods. We conduct experiment iteratively and for each iteration we prune 256 channels.of doing pruning and hints only once (i.e. First do pruning and then do hints. Both operations are conducted only once.) and doing pruning and hints iteratively. TAB2 shows . We can see that iterative operation can improve the performance of model dramatically. To further explain this , we do another experiment: we analyze the relationship between the performance of pruned model and the number of pruned channels. Results in FIG3 illustrate that when the number of pruned channels is large, the performance of pruned model will drop rapidlly. Thus, if we only do pruning and hints once, pruning will bring large degradation of performance and pruned model cannot output the similar feature to original model's. And in this way, pruned model is not a more resonable initialization to student model. Pruning step is useless to hints step in this situation. The teacher model is the pruned model from previous iteration in PWH Framework. Original model at the beginning of training can be another choice for teacher model in each iteration. We do an experiment to compare these two set-up for teacher model. And in this experiment, we prune 256 channels in each iteration. FIG4 shows . We observe that when iteration is small, using original model in the first iteration as teacher model has a comparable performance with using the pruned model in the previous iteration. However, with the increase of iterations, we can find that superiority of using the pruned model in the previous iteration increases. The reason is that the original model in the first iteration has higher accuracy so it performs well when iteration is small. But when iteration becomes large, pruned model's feature map will have large difference with original model's feature map. And in this situation, there is a gap between pruned model and teacher model in hints step. On the other hand, using the pruned model in the previous iteration will increase the similarity of feature map between student model's and teacher model's, which will help distillation in hints step. Proposed in §3.3, reconstructing step is used to further refine pruned model's feature and make it more similar to teacher's. We conduct the experiment to validate the effectiveness of reconstructing step. To fairly compare, we implement PWH Framework with and without reconstructing step. In each iteration, we prune 256 channels. We study on the accuracy of compressed model using two different methods. Furthermore, we also analyze L2 loss between pruned model's and original model's feature map in each iteration. Figure 5 shows experiment . We find that the method with reconstructing step performs better. We want our framework to be adaptive to different pruning methods but some of the pruning methods' criteria are not minimizing the reconstructing feature map's error. Reconstructing step can be used to solve this problem and increase the similarity of feature maps between two models. To further analyze the properties of PWH Framework, we conduct further experiments on our approach. The experiments verify our assumptions: pruning and hints are complementary to each other. All experiments are conducted on CIFAR-10 dataset using VGG16 as the original model. We conduct experiment to compare the reconstructing feature map error between pruned model and random initial model. We use the pruning method in §3.1 to prune certain number of channels from original network and we calculate L2 loss between pruned model's feature map and original model's feature map. Similarly, we randomly initialize a model whose size is same to the pruned model and calculate L2 loss of feature map between this model and original model. Then we increase pruned channel number and record these two errors accordingly. In TAB2, we notice that in a large range (0-1024 pruned channels) pruned model's feature map is much more similar to original model's. And this demonstrate that the transferability between pruned model and original model is larger. And student model, initialized with the weights of pruned model, can perform better in hints learning. To demonstrate hints method is beneficial to pruning, we first compare experiments between pruning-only with pruning and hints. Different from PWH Framework, pruning and hints method used in this section doesn't have reconstructing step. This is because we want to show the effectiveness of hints and reconstructing step is an extraneous variable. In contrast experiments, we iteratively implement pruning and hints operations. To fairly compare, in pruning and hints method, we substitute finetune operation for hints operation to get pruning-only method. In Figure 6, we observe that pruning and hints method has comparable performance with pruning-only method on small amount of iteration. However, the margin of two methods becomes larger and larger with the increase of iterations (pruned channels number). This phenomenon caused by the huge performance degradation in pruning operation when the original model is small. The small model doesn't have many redundant neurons and the main structure will be broken in pruning. And hints can alleviate this trend and help reconstruct parameters in pruned model. In this paper, we propose PWH Framework, an iterative framework for model acceleration. Our framework takes the advantage of both pruning and hints methods. To our best knowledge, this is the first work that combine these two model acceleration methods. Furthermore, we conduct reconstructing operation between hints and pruning steps as a cascader. We analyze the property of these two methods and show they are complementary to each other: pruning provides a better initialization for student model and hints method helps to adjust parameters in pruned model. Experiments on CIFAR-10, ImageNet and COCO datasets for classification and pose estimation tasks demonstrate the superiority of PWH Framework. In this supplementary material, we first provide more implementation details for better illustration of our experiments. In the second part, we give a proof in §3.3. We show that the upper bound of normalized L2 loss will decrease if L2 loss decreases theoretically. Following section contains more implementation details and We use PyTorch deep learning framework with 4 NVIDIA Titan X GPUs. A.0.1 CIFAR-10:CIFAR-10 dataset has 10 classes containing 6000 32 × 32 color images each. 50000 images are used for training and 10000 for test. We use top-1 error for evaluation. For model, we use the VGG-16 BID33 network with BatchNorm BID18. In CIFAR-10 finetune (hints) step, we use standard data augmentation with random cropping with 4 pixels, mean substraction of (0.4914, 0.4822, 0.4465) and std to (0.2023, 0.1994, 0.2010). A batch size of 128 and learning rate of 1e-3 are used. We set finetune (hints) epoch as 20. The hints method utilized on this dataset is L2 hints method with loss weight 10. We sample 1000 images to reconstruct weights. ImageNet classification dataset consists of 1000 classes. We train models on 1.28 million training images and test models on 100k images. Top-1 error is used to evaluate models. In this experiment, ResNet18 is our original model and we use PWH Framework to compress it. During finetune (hints) stage, the batchsize is 256, learning rate is 1e-3. We set mean substraction to (0.485, 0.456, 0.406) and std to (0.229, 0.224, 0.225). The loss weight is set as 0.5. The random cropping is used. In rescontructing stage, 1000 images are sampled to reconstruct hints layer weights. We conduct pose estimation experiment on COCO dataset. In this experiment, we train our models on trainval dataset and evaluate models on minival set. The evaluation criteria for COCO dataset we use is OKS-based mAP. We use ResNet18 with as the original model in the experiment. And in this experiment, we use random cropping, random rotation and random scale as our data augmentation strategy. We use a weight decay of 1e-5 and learning rate of 5e-5. The loss weight is set as 0.5. A batch size of 96 is used. The number of sampled images in reconstructing step is 500. In reconstructing step, we use least square method to reconstruct parameters in hints layer. The objective function of this step can be described in equation 7. DISPLAYFORM0 Where Y is the feature map of original (teacher) model, X is the input of hints layer and W is the parameter of hints layer. However, many hints methods use normalized L2 loss as hints loss (See equation 8). It's difficult to optimize the problem using common methods if we set normalized L2 loss as objective function. DISPLAYFORM1 In this section, we will show that the upper bound of normalized L2 loss is related to L2 loss. In other words, if L2 loss decreases, the upper bound of L2 loss will decrease. For an image x, y is the feature map of teacher network whose input is x. We suppose that: y = W x+e = y 0 + e DISPLAYFORM2 Where e is the error and it is independent with x. E[·] means the expectation. We suppose that 1 y can be expressed as the function of y 0 and e using taylor expansion. Where I is the identity matrix. For convenience, we assume that K =

Represent the TLDR for finding the corresponding scientific article.

This is a work aiming for boosting all the existing pruning and mimic method.

represent the tldr for finding the corresponding scientific article

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: For typical sequence prediction problems such as language generation, maximum likelihood estimation (MLE) has commonly been adopted as it encourages the predicted sequence most consistent with the ground-truth sequence to have the highest probability of occurring. However, MLE focuses on once-to-all matching between the predicted sequence and gold-standard, consequently treating all incorrect predictions as being equally incorrect. We refer to this drawback as {\it negative diversity ignorance} in this paper. Treating all incorrect predictions as equal unfairly downplays the nuance of these sequences' detailed token-wise structure. To counteract this, we augment the MLE loss by introducing an extra Kullback--Leibler divergence term derived by comparing a data-dependent Gaussian prior and the detailed training prediction. The proposed data-dependent Gaussian prior objective (D2GPo) is defined over a prior topological order of tokens and is poles apart from the data-independent Gaussian prior (L2 regularization) commonly adopted in smoothing the training of MLE. Experimental show that the proposed method makes effective use of a more detailed prior in the data and has improved performance in typical language generation tasks, including supervised and unsupervised machine translation, text summarization, storytelling, and image captioning. Language understanding is the crown jewel of artificial intelligence. As the well-known dictum by Richard Feynman states, "what I cannot create, I do not understand." Language generation therefore reflects the level of development of language understanding. Language generation models have seen remarkable advances in recent years, especially with the rapid development of deep neural networks (DNNs). There are several models typically used in language generation, namely sequenceto-sequence (seq2seq) models (; ; ; ;), generative adversarial networks (GANs) , variational autoencoders , and auto-regressive networks . Language generation is usually modeled as a sequence prediction task, which adopts maximum likelihood estimation (MLE) as the standard training criterion (i.e., objective). MLE has had much success owing to its intuitiveness and flexibility. However, sequence prediction has encountered the following series of problems due to MLE. • Exposure bias: The model is not exposed to the full range of errors during training. • Loss mismatch: During training, we maximize the log-likelihood, whereas, during inference, the model is evaluated by a different metric such as BLEU or ROUGE. • Generation diversity: The generations are dull, generic (; ; a), repetitive, and short-sighted (b). • Negative diversity ignorance: MLE fails to assign proper scores to different incorrect model outputs, which means that all incorrect outputs are treated equally during training. A variety of work has alleviated the above MLE training shortcomings apart from negative diversity ignorance. Negative diversity ignorance is a of unfairly downplaying the nuance of sequences' detailed token-wise structure. When the MLE objective compares its predicted and ground-truth sequences, it takes a once-for-all matching strategy; the predicted sequence is given a binary label, either correct or incorrect. However, these incorrect training predictions may be quite diverse and letting the model be aware of which incorrect predictions are more incorrect or less incorrect than others may more effectively guide model training. For instance, an armchair might be mistaken with a deckchair, but it should usually not be mistaken for a mushroom. To alleviate the issue of the negative diversity ignorance, we add an extra Gaussian prior objective to augment the current MLE training with an extra Kullback-Leibler divergence loss term. The extra loss is computed by comparing two probability distributions, the first of which is from the detailed model training prediction and the second of which is from a ground-truth token-wise distribution and is defined as a kind of data-dependent Gaussian prior distribution. The proposed data-dependent Gaussian prior objective (D2GPo) is then injected into the final loss through a KL divergence term. The D2GPo is poles apart from the commonly adopted data-independent Gaussian prior (L2 regularization) for the purpose of smoothing the training of MLE, which is also directly added into the MLE loss. Experimental show that the proposed method makes effectively use of a more detailed prior in the data and improves the performance of typical language generation tasks, including supervised and unsupervised machine translation, text summarization, storytelling, and image captioning. Natural language generation (NLG) has long been considered the most challenging natural language processing (NLP) task . NLG techniques have been widely adopted as a critical module in various tasks, including control-free sentence or poem generation and input-conditioned language generation, such as machine translation, image captioning, text summarization, storytelling (; ; ;), and sentiment/tense-controlled sentence generation. In this work, we focus on input-conditioned language generation tasks, though our proposed method can also be applied to other language generation fields. Input-conditioned language generation tasks are challenging because there is an information imbalance between the input and output in these tasks, especially for cases with non-text input . discussed different ways of building complicated knowledge-based systems for NLG. In recent years, neural networks (NNs), especially DNNs, have shown promising in many NLP tasks. first proposed the NN language model (NNLM) to exploit the advantages of NNs for language generation tasks. In an NNLM, the n-gram paradigm is extended by the generalization ability of NNs. Given the ground truth sequence s = w 1, w 2,..., w t−1, the NNLM adopts the equation p t ≈ p(w t |w t−n, w t−n+1, ..., w t−1). developed a more general implementation for a language model (called the recurrent NN language model (RNNLM) by integrating a Markov property using a recurrent NN (RNN) to address the NNLMs' theoretical inability to capture long-term dependencies: The RNNLM is an effective solution because it is designed to capture long-term dependencies. Because of the vanishing gradient problem in RNNs, however, the long-term dependency processing capability is limited. In contrast to an RNN, the Transformer provides a new self-attention mechanism for handling long-term dependencies in text, ing in robust performance across diverse tasks. proposed a Transformer language model called GPT, which uses a left-to-right architecture, where each token pays attention to previous tokens in the self-attention layers of the Transformer. introduced a new pre-training objective: the masked language model (MLM), which enables the representation to fuse the left and right contexts and allows us to pre-train a deep bidirectional Transformer called BERT. The generators of the most current language generation model use the RNNLM or Transformer language model structure. However, as pointed out by, fitting the distribution of observation data does not mean that satisfactory text will be generated, because the model is not exposed to the full range of errors during training. This is called the exposure bias problem. Reinforcement learning, GANs , and end-to-end re-parameterization techniques (Kusner & Hernández-) have been proposed to solve this problem. The exposure bias is no longer an issue in reinforcement learning models because the training sequences are generated by the model itself. Using MLE for the training objective leads to the problem of loss mismatch. incorporated the evaluation metric into the training of sequence-to-sequence (seq2seq) models and proposed the mixed incremental cross-entropy reinforce (MIXER) training strategy, which is similar to the idea of minimum risk training (; ; ; There is an increasing interest in incorporating problem field knowledge in machine learning approaches (; ;). One common way is to design specialized network architectures or features for specific knowledge (e.g.,). In contrast, for structured probabilistic models, posterior regularization and related frameworks (; ;) provide a general means to impose knowledge constraints during model estimation. established a mathematical correspondence between posterior regularization and reinforcement learning and, using this correspondence, expanded posterior regularization to learn knowledge constraints as the extrinsic reward in reinforcement learning. Our approach can be seen as incorporating a priori knowledge of the language field into language generation learning. proposed a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. The difference is that focuses on low-frequency words while our model focuses on negative tokens. It is claimed that the likelihood objective itself is at fault, ing in a model that assigns too much probability to sequences containing repetition and frequent words, unlike those from the human training distribution. From this point of view, there is a point in common with our motivation, which is to make the model prediction consistent with human training distribution to some extent. Consider a conditional probability model for sequence predictions y ∼ p θ (x) with parameters θ. The target sequence y can be conditioned on any type of source x (e.g., a phrase, sentence, or passage of human language or even an image), which is omitted for simplicity of notation. For the sequence y = y 1, y 2,..., y l, the probability p θ (y|x) is Published as a conference paper at ICLR 2020 Commonly, sequence prediction models are trained using MLE (also known as teacher forcing) . MLE minimizes the negative log-likelihood of p θ (y|x): Optimizing the MLE objective L MLE (θ) is straightforward and meets the principle of empirical risk minimization while focusing on only minimizing losses of the correct target on the training data set. However, there may be noise in the training data, and forcibly learning the distribution of a training set does not enable the obtained model to reach good generalization. Additionally, for sequence prediction, models trained subject to MLE cursorily evaluate all predictions as either correct or incorrect and ignore the similarity between the correct and "less incorrect" predictions. Incorrect predictions might range from nearly perfect (i.e., one token is mistaken with a synonym) to completely wrong, having nothing in common with the gold sequence. However, MLE training treats all incorrect training predictions equally, which implies that MLE fails to accurately assign scores to diverse (especially negative) model predictions. To capture the diversity of negative training predictions, we augment the MLE objective of the model with an additional objective O that more accurately models such a negative diversity. Without loss of generality, supposingỹ is the prediction candidate, we introduce a general evaluation function f (ỹ, y) ∈ R independent of the model prediction, such that with a golden target token y *, a higher f (ỹ, y *) value indicates a better p θ (ỹ|x) for a target candidateỹ ∈ V (where V is the target candidate set). Note that f (ỹ, y) can also involve other factors such as latent variables and extra supervisions. There are two main methods of learning f (ỹ, y) in the model. If p θ is a GAN-like implicit generative model or an explicit distribution that can be efficiently reparametrized (e.g., Gaussian) , then one effective method is maximizing E p θ [f (ỹ, y)]. The other method is computing the gradient ∇ θ E p θ [f (ỹ, y)] using the log-derivative trick that can suffer from high variance but is often used for the large set of non-parameterizable explicit distributions. Corresponding to the probability distribution of model predictions p θ (·), we define a prior distribution q(y) (for each target y i, it has its own unique distribution of q i = q(y i)) which is extracted and derived from the ground-truth data (e.g., language text in language generation tasks). To guide the probability distribution of model predictions p θ (·) to match the prior probability distributions q(·), we adopt Kullback-Leibler divergence. Considering also the learning of the evaluation function f (ỹ, y), the loss for objective O is calculated as where α is a weight for the evaluation function learning term. We derive the prior distribution q(y) from the ground-truth data (which is independent of model parameters θ), and therefore in which KL divergence can be expanded as The final objective for learning the model is written as where λ is the balancing hyperparameter. Because optimizing the original model objective L MLE (θ) is straightforward, in the following, we omit the discussion of L MLE (θ) and focus on the proposed The prior probability distribution q(y *) on y * can be obtained from the evaluation function f (·, ·) with a softmax operation. To expose the mass of the distribution over the classes, introduced a softmax temperature mechanism; therefore, the relationship between q and f (ỹ, y) is where T is a temperature parameter. When T → 0, the distribution becomes a Kronecker distribution (and is equivalent to a one-hot target vector); when T → +∞, the distribution becomes a uniform distribution. The softmax operation always turns an evaluation function f (·, ·) into a form of probability distribution no matter the form of the original f (·, ·); thus, we only focus on f (·, ·). To find a good evaluation function, we have to mine token-wise diversity for all y *. Considering all token typesỹ j in a vocabulary, with respect to each y *, there exists a prior topological order ORDER(y *) among all the known tokens, in which y * is always ranked top priority. f (ỹ j, y *) can then be defined as a monotonic function over the corresponding topological order so that it has a maximal value only when the input is y * itself. Note that defining f (·, ·) in this way leads to the ing q also being monotonic over the corresponding topological order. Considering that q is a priori, it is fixed throughout the learning process. The remaining questions are about how to find a meaningful evaluation function f (·, ·) for the distribution q. In language generation tasks, we may conveniently take word embedding as the token representation, and let the embedding distance determine such an order ORDER(y *) for each y *. In this work, we adopt the cosine similarity of pre-trained embeddings to sort the token (word/subword) order. Discussion For the evaluation function f (·, ·) of q, we adopt the Gaussian probability density function, though later we also present experimental for other types of functions in an ablation study. As the adopted Gaussian prior used in the training objective is derived from a datadependent token-wise distribution, we call it the data-dependent Gaussian prior objective (D2GPo). This objective is a big departure from the Gaussian prior commonly adopted for smoothing in MLE training (which we the data-independent Gaussian prior). The following briefly explains why we chose the Gaussian probability density function and how our D2GPo mathematically differs from the data-independent Gaussian prior. The central limit theorem indicates that suitably standardized sums of independent random variables have an approximately normal distribution. Thus, any random variable that arises as the sum of a sufficiently large number of small random components can be modeled accurately using a normal distribution. Embedding has a linear additive property (e.g., king -man + woman ≈ queen). The additive property of embedding can be explained by inspecting the training objective . Each dimension of an embedding represents a potential feature of the token. Considering each potential feature as an independent random variable, the sum follows a Gaussian distribution centered on the correct vocabulary unit y * according to the linear additive property. We can therefore use a Gaussian distribution for the embedding-distance-determined order to effectively model the distribution q(y *). An overview of the concepts underlying D2GPo is illustrated in Appendix A.1. The D2GPo in this paper is different from the data-independent Gaussian prior in machine learning optimization theory. We hypothesize and experimentally verify that the embedding feature extracted from the data obeys the Gaussian distribution. The distribution from the prior knowledge of language data is used as a soft target to guide the model language generation process using knowledge distillation. The Gaussian prior in the machine learning optimization theory assumes that each component in the parameter θ is subject to a zero-mean Gaussian prior distribution, which is equivalent to L2 regularization. In general, our Gaussian prior objective is to act on the guiding target probability, while the Gaussian prior in machine learning is applied to the selection of model parameters. This section describes the experimental evaluation of D2GPo on a variety of typical language generation tasks: neural machine translation (NMT), text summarization, storytelling, and image captioning. The hyperparameters in D2GPo and effect analysis are given in Appendix A.7. Our proposed D2GPo approach for experimental tasks requires either word embeddings or bytepair-encoding (BPE) (b) subword embeddings. We generated the pretrained embeddings using fastText with an embedding dimension of 512, a context window of size 5, and 10 negative samples. For NMT, fastText was applied to the concatenation of source and target language monolingual corpora, ing in cross-lingual BPE subword embedding. For text summarization, we generated BPE subword embedding only on the English monolingual corpora, while for the storytelling and image captioning, we obtained the word embedding also on the English monolingual corpora. We evaluated the model on several widely used translation tasks: WMT14 English-to-German (EN-DE), English-to-French (EN-FR), and WMT16 English-to-Romanian (EN-RO) 1 tasks, which all have standard large-scale corpora for NMT evaluation. Owing to space limits, the data details are provided in Appendix A.3. The sentences were encoded using sub-word types based on BPE, which has a shared vocabulary of 40,000 sub-word units for all three tasks. We chose the Transformer NMT model as our baseline. For the hyperparameters of the Transformer (base/big) models, we followed the settings used by. The BLEU score with multi-bleu.pl was calculated during the evaluation. Table 1: Comparison with baseline and existing systems on supervised translation tasks. Here, "++/+" after the BLEU score indicates that the proposed method was significantly better than the corresponding baseline Transformer (base or big) at significance levels p <0.01/0.05. "STD" represents synthetic training data from (b). In Table 1, we report the performance of our full model, the baseline, and existing systems. Our baseline model obtains similar to those of , the existing strong model used for these tasks. The indicate that our method performed better than the strong baselines for all language pairs. Our model is not only an improvement on the translation model of large-scale training sets but also performs better for small-scale training sets. Refer to Appendix A.8 and A.9 for analysis of the low-resource scenario and generation diversity. For unsupervised machine translation, we also used the three language pairs EN-DE, EN-FR, and EN-RO as our evaluation targets. Note that the evaluation performed on EN-DE uses newstest2016 instead of newstest2014 to ensure the are comparable with the of other works; this is unlike supervised machine translation. We used the masked sequence to sequence the pre-training (MASS) model as our baseline. Following the practice of , we pretrained our model with a masked sequence-to-sequence pre-training (MASS) objective (without D2GPo) on EN, FR, DE, and RO monolingual data samples from WMT 2007-2018 News Crawl datasets that respectively cover 190M, 60M, 270M, and 10M sentences. We then fine-tuned the models on the same monolingual data using the back-translation cross-entropy loss and our D2GPo loss. For the training dataset, we filtered out sentences longer than 175 words in length and jointly learned 60K BPE sub-word units for each language pair. Table 2: BLEU score comparisons between MASS and previous methods of unsupervised NMT. As shown in Table 2, D2GPo consistently outperformed MASS (the state-of-the-art baseline) on all unsupervised translation pairs. Meanwhile, the MASS and XLMsystems leverage large-scale monolingual pre-training, and the decoder (generator, language model) can still be improved by our D2GPo loss in the fine-tuning phase. This demonstrates the efficiency of the proposed method. Text summarization is a typical language generation task that creates a short and fluent summary of the given long-text document. fine-tuned the MASS pretrained model on the text summarization task and achieved state-of-the-art . We chose this model as our baseline, maintained consistent pre-training, and used D2GPo loss for enhancements in the fine-tuning phase. We used the Annotated Gigaword corpus as the benchmark, as detailed in Appendix A.4. In the evaluation, ROUGE-1, ROUGE-2, and ROUGE-L are reported. Table 3. We compared our +D2GPo with our baseline MASS, which is the current state-of-the-art model; +D2GPo consistently outperformed the baseline on all evaluation metrics. The models with a semi-supervised setting yielded a large-margin improvement relative to the model without any pre-training, which demonstrates that the supervised pre-training is effective in the text summarization task. Storytelling is at the frontier of current language generation technologies; i.e., stories must maintain a consistent theme throughout and require long-distance dependency modeling. Additionally, stories require creativity and a high-level plot with planning ahead rather than word-by-word generation . We used the hierarchical story generation model (which is introduced in Appendix A.5) as our baseline to test the improvements of D2GPo for the storytelling task. To guarantee the single-variable principle, we added only the D2GPo loss to the story generation model. The prompt generation model is consistent with. For automatic evaluation, we measured the model perplexity on validation and test sets. Table 4 shows obtained using D2GPo. It is seen that with the addition of D2GPo, the Conv seq2seq + self-attention model substantially improved the likelihood of human-generated stories and even outperformed the ensemble or fusion models without increasing the number of parameters. Perplexity was further reduced with the addition of the fusion mechanism. These suggest that Table 4: Perplexity on WRITINGPROMPTS. D2GPo improves the quality of language generation greatly, especially in settings where there are fewer restrictions on story generation tasks. Image captioning is a task that combines image understanding and language generation. It continues to inspire considerable research at the boundary of computer vision and natural language processing. We elected to experiment with image captioning to verify the performance of D2GPo on a language generation model having diverse types of input. In our experiments, we evaluated our model on an ablated baseline (top-down, as detailed in Appendix A.6) against prior work on the MSCOCO 2014 caption dataset , which has became the standard benchmark for image captioning. For validation of model hyperparameters and offline testing, we used Karpathy splits , which have been used extensively in prior work. SPICE , CIDEr , METEOR , ROUGE-L, and BLEU were used to evaluate the caption quality. Table 5: Image caption performance on the MSCOCO Karpathy test split. Table 5 summarizes the performance of our full model and the ResNet Top-down baseline in comparison with the existing strong Self-critical Sequence Training (SCST) approach on the test portion of the Karpathy splits. To ensure a fair comparison, are only reported for models trained with standard cross-entropy loss (i.e., MLE). All are reported for a single model with no fine tuning of the input ResNet model. Our ResNet baseline performs slightly better than the SCST models. After incorporating our proposed D2GPo loss, our model improves further across all metrics. According to the analysis in Section 4, for the embedding, we used the Gaussian probability density function as our evaluation function f (·); however, to evaluate the effectiveness of different evaluation functions, we changed the function and tested the performance changes on the supervised NMT EN-DE task. We used the same experiment settings as described in Section 5.2 and compared the BLEU score changes on the test set, as listed in Table 6. Table 6: Ablation study on our proposed D2GPo with different evaluation functions on the supervised NMT WMT14 EN-DE task, with the Transformer-base model. The table shows that the performance of Gaussian density, linear, and cosine functions increased while the performance of the random function decreased. This shows that the distance information obtained from embedding can effectively guide the generation process. Among these functions, the Gaussian density function had the greatest improvement, which agrees with our analysis of the embedding features obeying the Gaussian distribution. We postulate that because the linear and cosine functions are rough approximations of the Gaussian density function, they perform similarly to the Gaussian density function. This work proposed a data-dependent Gaussian prior objective (D2GPo) for language generation tasks with the hope of alleviating the difficulty of negative diversity ignorance. D2GPo imposes the prior from (linguistic) data over the sequence prediction models. D2GPo outperformed strong baselines in experiments on classic language generation tasks (i.e., neural machine translation, text summarization, storytelling, and image captioning tasks). A.1 CONCEPTS UNDERLYING D2GPO Figure 1: Overview of the concepts underlying D2GPo taking the example of the sentence The little boy sits on the armchair. Specifically, for target y i, we calculate the embedding cosine similarity as the distance dist(i, j) of y i and all other token types in the vocabularyỹ j, which are used to give the distance: dist i,j = cosine similarity(emb(y i), emb(ỹ j)). Sorting by distance from small to large to obtain the topological order of token types yields A.3 SUPERVISED NMT DATA For the EN-DE translation task, 4.43M bilingual sentence pairs from the WMT'14 dataset, which includes the Common Crawl, News Commentary, and Europarl v7 datasets, were used as training data. The newstest2013 and newstest2014 datasets were used as the dev set and test set, respectively. For the EN-FR translation task, 36M bilingual sentence pairs from the WMT'14 dataset were used as training data. The newstest2012 and newstest2013 datasets were combined for validation and newstest2014 was used as the test set, following the configuration of. For the EN-RO task, we tested two settings; i.e., Europarl v7, which uses only the officially provided parallel corpus, and SETIMES2, which yields 600,000 sentence pairs for a low-resource supervised machine translation study. Alternatively, following the work of Sennrich et al. (2016a), we used synthetic training data (STD) of Sennrich et al. (2016a), which provides 2.8M sentence pairs for training. We used newsdev2016 as the dev set and newstest2016 as the test set. Our reported on EN-RO are evaluated on a reference for which diacritics are removed from letters. The Annotated Gigaword corpus was used as a benchmark . This data set is derived from news articles and comprises pairs of main sentences in the article (longer) and headline (shorter). The article and headline were respectively used as the source input sentence and reference. The data include approximately 3.8M training samples, 400,000 validation samples, and 2000 test samples. The hierarchical story generation model was proposed for the situation in which a sentence called a prompt that describes the topic of the upcoming story generation is first generated, and then conditions on the prompt are applied when generating the story. used a self-attention gated convolutional language model (GCNN) as the sequence-to-sequence prompt generation model with top-k random sampling. For prompt-tostory generation, they collected a dataset from Reddit's WRITINGPROMPTS forum in which each prompt has multiple story responses. With the dataset, they trained a story generation model that benefitted from a novel form of model fusion that improved the relevance of the story to the prompt and added a new gated multi-scale self-attention mechanism to model the long-range context. The top-down image captioning model uses a ResNet convolutional neural net pretrained on ImageNet to encode each image. Similar to previous work , the cited study encoded the full-sized input image with the final convolutional layer of Resnet-101 and used bilinear interpolation to resize the output to a fixed-size spatial representation of 10×10. This is equivalent to the maximum number of spatial regions used in our full model. During training with our D2GPo, the value of the standard deviation of the KL diversity item λ was set to 0.1, and the softmax temperature was T = 2.0 in all experiments. To study the effects of hyperparameters (i.e., the standard deviation λ and softmax temperature T The experimental reveal that λ affects the model training process. We believe that the reason is that a small value of λ in the model being unable to make full use of the prior knowledge (distribution), while a larger value of λ will make the model more uncertain because of the higher probability of there being incorrect or even opposite words whose fastText embeddings are similar. In addition, experimental show that a small value of T can improve the model to some extent, whereas a large value of T will seriously decrease the performance of the model. Theoretically, when T approaches infinity, the distribution q becomes uniform, and there is no prior knowledge with which to guide the model. A loss penalty is applied to any model prediction, and an excessively high value of T is thus harmful to training. Priors are generally more helpful in low-data regimes. We sampled 10,000, 100,000, and 600,000 paired sentences from the bilingual training data of WMT16 EN-RO to explore the performance of D2GPo in different low-resource scenarios. We used the same BPE code and learned fastText embeddings in all WMT16 EN-RO training data. Table 7: Comparison of our baseline and our D2GPo method under different training data scales in terms of BLEU on the WMT16 EN-RO test set. As shown in Table 7, D2GPo outperforms the baseline model, demonstrating the effectiveness of our method in low-resource scenarios. At the same time, the show that the performance improvement provided by D2GPo increases with fewer training data. This shows that prior knowledge can substantially improve the performance of the model when training data are scarce. A possible reason is that the training data are insufficient to train a robust model. In this case, the injection of prior knowledge can help train the parameters of the model and substantially improve the translation performance. However, with an increase in the number of training data, the model itself can be optimized well, and the improvement gained by introducing prior knowledge is not as substantial as before. Compared with traditional MLE training, D2GPo encourages negative diversity. To examine differences between D2GPo and MLE models, we counted high-and low-frequency words in the training set and compared the frequencies of low-frequency words predicted by the two models and the golden reference on the test set. Table 8: The statics of low frequency words in the reference and generations. The experiment was carried out on WMT14 EN-DE, the baseline model was Transformer-base, and the statistics were calculated at the word level. We chose words with a frequency less than or equal to 100 in the training set as low-frequency words. We used the golden reference (#GOLD), baseline model prediction output, and +D2GPo model prediction output to count the total number of tokens (#SUM) and the number of low-frequency words (#LF). Results are given in Table 8. The show that compared with the baseline, the D2GPo optimized model generates more lowfrequency words and has a higher ratio of low-frequency words. However, the number is still far less than the golden reference. It is thus demonstrated that D2GPo increases the diversity of model output. Top-down: a woman holding an umbrella in her hand + D2GPo: a woman is holding an umbrella + SCST: a woman holding an umbrella in a street + SCST+ D2GPo: a woman is holding an umbrella in the street Top-down: a large airplane sitting on top of an airport runway + D2GPo: an airplane is sitting on top of an airport runway + SCST: a large jetliner sitting on top of an airport runway + SCST+ D2GPo: a large jetliner is sitting on top of an airport runway Top-down: a woman holding a surf board in the ocean + D2GPo: a woman is standing on the beach with a surfboard + SCST: a woman holding a surfboard on the beach + SCST+ D2GPo: a woman is standing on the beach with a surfboard Top-down: a traffic light with a traffic light on it + D2GPo: a traffic light on the side of a traffic light + SCST: a yellow traffic light on the side of a street + SCST+ D2GPo: yellow traffic lights on the side of a street Table 9: Captions generated for the left image by the various models described in the paper. The models trained with SCST return a more accurate and more detailed summary of the image. The models trained with D2GPo return a more grammatically complete sentence.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce an extra data-dependent Gaussian prior objective to augment the current MLE training, which is designed to capture the prior knowledge in the ground-truth data.

Represent the short summary for finding the corresponding scientific article (long):

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.