Question: Represent the article for finding a single-sentence TLDR

Question: A belief persists long in machine learning that enlargement of margins over training data accounts for the resistance of models to overfitting by increasing the robustness. Yet Breiman shows a dilemma that a uniform improvement on margin distribution \emph{does not} necessarily reduces generalization error. In this paper, we revisit Breiman's dilemma in deep neural networks with recently proposed normalized margins using Lipschitz constant bound by spectral norm products. With both simplified theory and extensive experiments, Breiman's dilemma is shown to rely on dynamics of normalized margin distributions, that reflects the trade-off between model expression power and data complexity. When the complexity of data is comparable to the model expression power in the sense that training and test data share similar phase transitions in normalized margin dynamics, two efficient ways are derived via classic margin-based generalization bounds to successfully predict the trend of generalization error. On the other hand, over-expressed models that exhibit uniform improvements on training normalized margins may lose such a prediction power and fail to prevent the overfitting. Margin, as a measurement of the robustness allowing some perturbations on classifier without changing its decision on training data, has a long history in characterizing the performance of classification algorithms in machine learning. As early as BID17, it played a central role in the proof on finite-stopping or convergence of perceptron algorithm when training data is separable. Equipped with convex optimization technique, a plethora of large margin classifiers are triggered by support vector machines BID3 BID23. AdaBoost, an iterative algorithm to combine an ensemble of classifiers proposed by BID4, often exhibits a resistance to overfitting phenomenon that during the training process the generalization error keeps on non-increasing when the training error drops to zero. Toward deciphering the such a resistance of overfitting phenomenon, BID19 proposed an explanation that the training process keeps on improving a notion of classification margins in boosting, among later works on consistency of boosting with early stopping regularization BID2 BID30 BID28. Lately such a resistance to overfitting is again observed in deep neural networks with overparameterized models. A renaissance of margin theory is proposed by BID0 with a normalization of network using Lipschitz constants bounded by products of operator spectral norms. It inspires many further investigations in various settings BID14 BID16 BID12.However, the improvement of margin distributions does not necessarily guarantee a better generalization performance, which is at least traced back to BID1 in his effort to understanding AdaBoost. In this work, Breiman designed an algorithm arc-gv such that the margin can be maximized via a prediction game, then he demonstrated an example that one can achieve uniformly larger margin distributions on training data than AdaBoost but suffer a higher generalization error. In the end of this paper, Breiman made the following comments with a dilemma: "The above leave us in a quandary. The laboratory for various arcing algorithms are excellent, but the theory is in disarray. The evidence is that if we try too hard to make the margins larger, then overfitting sets in.... My sense of it is that we just do not understand enough about what is going on."Breiman's dilemma triggers some further explorations to understand the limitation of margin theory in boosting BID18; BID27. In particular, BID18 points out that the trees found by arg-gv have larger model complexity in terms of deeper average depth than AdaBoost, suggesting that margin maximization in arc-gv does not necessarily control the model complexity. The latter works provide tighter bounds based on VC-dimension and optimized quantile training margins, which however do not apply to over-parametrized models in deep neural networks and the case where the training margin distributions are uniformly improved. In this paper, we are going to revisit Breiman's dilemma in the scenario of deep neural networks. Both the success and failure can be seen on normalized margin based bounds on generalization error. First of all, let's look at the following illustration example. Example (Breiman's Dilemma with a CNN). A basic 5-layer convolutional neural network of c channels (see Section 3 for details) is trained with CIFAR-10 dataset whose 10 percent labels are randomly permuted. When c = 50 with 92, 610 parameters, FIG0 shows the training error and generalization (test) error in solid curves. From the generalization error in (a) one can see that overfitting indeed happens after about 10 epochs, despite that training error continuously drops down to zero. One can successfully predict such an overfitting phenomenon from FIG0 (b), the evolution of normalized margin distributions defined later in this paper. In (b), while small margins are monotonically improved during training, large margins undergoes a phase transition from increase to decrease around 10 epochs such that one can predict the tendency of generalization error in (a) using large margin dynamics. Two particular sections of large margin dynamics are highlighted in (b), one at 8.3 on x-axis that measures the percentage of normalized training margins no more than 8.3 (training margin error) and the other at 0.8 on y-axis that measures the normalized margins at quantile q = 0.8 (i.e. 1/γ q,t). Both of them meet the tendency of generalization error in (a) and find good early stopping time to avoid overfitting. However, as we increase the channel number to c = 400 with about 5.8M parameters and retrain the model, (c) shows a similar overfitting phenomenon in generalization error; on the other hand, (d) exhibits a monotonic improvement of normalized margin distributions without a phase transition during the training and thus fails to capture the overfitting. This demonstrates the Breiman's dilemma in CNN. A key insight behind this dilemma, is that one needs a trade-off between the model expression power and the complexity of the dataset to endorse margin bounds a prediction power. On one hand, when the model has a limited expression power relative to the training dataset, in the sense that the training margin distributions CAN NOT be uniformly improved during training, the generalization or test error may be predicted from dynamics of normalized margin distributions. On the other hand, if we push too hard to improve the margin by giving model too much degree of freedom such that the training margins are uniformly improved during training process, the predictability may be lost. A trade-off is thus necessary to balance the complexity of model and dataset, otherwise one is doomed to meet Breiman's dilemma when the models arbitrarily increase the expression power. The example above shows that the expression power of models relative to the complexity of dataset, can be observed from the dynamics of normalized margins in training, instead of counting the number of parameters in neural networks. In the sequel, our main contributions are to make these precise by revisiting the Rademacher complexity bounds with Lipschitz constants BID0.• With the Lipschitz-normalized margins, a linear inequality is established between training margin and test margin in Theorem 1. When both training and test normalized margin distributions undergo similar phase transitions on increase-decrease during the training process, one may predict the generalization error based on the training margins as illustrated in FIG0.• In a dual direction, one can define a quantile margin via the inverse of margin distribution functions, to establish another linear inequality between the inverse quantile margins and the test margins as shown in Theorem 2. Quantile margin is far easier to tune in practice and enjoys a stronger prediction power exploiting an adaptive selection of margins along model training.• In all cases, Breiman's dilemma may fail both of the methods above when dynamics of normalized training margins undergo different phase transitions to that of test margins during training, where a uniform improvement of margins in overfitting. Section 2 describes our method to derive the two linear inequalities of generalization bounds above. Extensive experimental are shown in Section 3 and Appendix with basic CNNs, AlexNet, VGG, ResNet, and various datasets including CIFAR10, CIFAR100, and mini-Imagenet. Let X be the input space (e.g. X ⊂ R C×W ×H in image classification) and Y:= {1, . . ., K} be the space of K classes. Consider a sample set of n observations S = {(x 1, y 1),..., (x n, y n): x i ∈ X, y i ∈ Y} that are drawn i.i.d. from P X,Y. For any function f: X → R, let Pf = X f (X)dP be the population expectation and P n f = (1/n) n i=1 f (x i) be the sample average. Define F to be the space of functions represented by neural networks, DISPLAYFORM0 where l is the depth of the network, W i is the weight matrix corresponding to a linear operator on x i and σ i stands for either element-wise activation function (e.g. ReLU) or pooling operator that are assumed to be Lipschitz bounded with constant L σi and satisfying σ i = 0. For example, in convolutional network, W i x i + b i = w i * x i + b i where * stands for the convolution between input tensor x l and kernel tensor w l. We equip F with the Lipschitz semi-norm, for each f, DISPLAYFORM1 where · σ is the spectral norm and DISPLAYFORM2 For all the examples in this paper, we use ReLU activation σ i that leads to L σi = 1. Moreover we consider the following family of hypothesis mapping, DISPLAYFORM3 where [·] j denotes the j th coordinate and we further define the following class induced by Lipschitz semi-norm bound on F, DISPLAYFORM4 Lastly, rather than merely looking at whether a prediction f (x) on y is correct or not, we also consider the margin defined as ζ(f (x), y) = [f (x)] y −max {j:j =y} [f (x)] j. Therefore, we can define the ramp loss and margin error depending on the confidence of predictions. Given two thresholds γ 2 > γ 1 ≥ 0, define a ramp loss to be DISPLAYFORM5 where ∆:= γ 2 − γ 1. In particular γ 1 = 0 and γ 2 = γ, we also write γ = γ for simplicity. Define the margin error to measure if f has margin no more than a threshold γ, DISPLAYFORM6 In particular, e 0 (f (x), y) is the common mis-classification error and DISPLAYFORM7 Note that e 0 ≤ γ ≤ e γ, and γ is Lipschitz bounded by 1/γ. The central question we try to answer is, can we find a proper upper bound to predict the tendency of the generalization error along training, such that one can early stop the training near the epoch that DISPLAYFORM8 The answer is both a yes and a no!We begin with the following lemma, as a typical in multi-label classification from the uniform law of large numbers BID8. Lemma 2.1. Given a γ 0 > 0, then, for any δ ∈, with probability at least 1 − δ, the following holds for any f ∈ F with f F ≤ L, DISPLAYFORM9 is the Rademacher complexity of function class H L with respect to n samples, and the expectation is taken over x i, ε i, i = 1,..., n. Unfortunately, direct application of such bound for a constant γ 0 will suffer from the so-called scaling problem. The following proposition gives an lower bound of Rademacher complexity term, whose proof is provided in Appendix D. Proposition 1. Consider the networks with ReLU activation functions. For any L > 0, there holds, DISPLAYFORM10 where C > 0 is a constant that does not depend on S.The lemma tells us if L → ∞, upper bound becomes trivial since R n (H L) → ∞. In fact, both BID22 and BID21 show that with gradient descent, the norm of estimator's weight in logistic regression and general boosting (including exponential loss), respectively, will go to infinity at a growth rate log(t) when the data is linearly separable. As for the deep neural network with cross-entropy loss, the input of last layer is usually be viewed as features extracted from original input. Training the last layer with other layers fixed is exactly a logistic regression, and the feature is linearly separable as long as the training error achieves zero. Therefore, without any normalization, the hypothesis space along training has no upper bound on L and the upper bound is useless. Besides, even for a fixed L, the complexity term R n (H L) is computationally intractable. The first remedy is to restrict our attention on H 1 by normalizing f with its Lipschitz semi-norm f F or its upper bounds. Note that a normalized networkf = f /C has the same mis-classification error as f for all C > 0. For the choice of C, it's hard in practice to directly compute the Lipschitz semi-norm of a network, but instead some approximate estimates on the upper bound L f in are available as discussed in Appendix A. In the sequel, letf = f /L f be the normalized network and DISPLAYFORM11 be the corresponding normalized hypothesis function. Now a simple idea is to regard R n (H 1) as a constant and predict the tendency of generalization error via training margin error of the normalized network, that avoids the scaling problem and the computation of complexity term. The following theorem makes this precise. Theorem 1. Given γ 1 and γ 2 such that γ 2 > γ 1 ≥ 0 and ∆:= γ 2 − γ 1 ≥ 0, for any δ > 0, with probability at least 1 − δ, along the training epoch t = 1,..., T, the following holds for each f t, DISPLAYFORM12 where DISPLAYFORM13 Remark. In particular, when we take γ 1 = 0 and γ 2 = γ > 0, the bound above becomes, DISPLAYFORM14 Theorem 1 says, we can bound the normalized test margin distribution DISPLAYFORM15 Recently BID12 investigates for normalized networks, the strong linear relationship between cross-entropy training loss and test loss when the training epochs are large enough. As a contrast, we consider the whole training process and normalized margins. In particular, we hope to predict the trend of generalization error by choosing γ 1 = 0 and a proper γ. For this purpose, the following facts are important. First, we do not expect the bound, for example, is tight for every choice of γ > 0, instead we hope there exists some γ such that the training margin error nearly monotonically changes with generalization error. FIG1 shows the existence of such γ such that the training margin error successfully recover the tendency of generalization error on CIFAR10 dataset. Moreover, in Appendix Figure 8 shows the rank correlation between training margin error at various γ and training/test error. Second, the normalizing factor is not necessarily to be an upper bound of Lipschitz semi-norm. The key point is to prevent the complexity term of the normalized network going to infinity. Since for any constant c > 0, normalization byL = cL works in practice where the constant could be absorbed to γ, we could ignore the Lipschitz constant introduced by general activation functions in the middle layers. However, it is a natural question whether a reasonable γ with prediction power exists. A simple example in FIG0 shows, once the training margin distribution is uniformly improved, dynamic of training margin error fails to detect the minimum of generalization error in the early stage. This is because when network structure becomes complex enough, the training margin distribution could be more easily improved but the the generalization error may overfit. This is exactly the same observation in BID1 to doubt the margin theory in boosting type algorithms. More detailed discussions will be given in Section 3.2.The most serious limitation of Theorem 1 lies in we must fix a γ along the complete training process. In fact, the first term and second term in the bound vary in the opposite directions with respect to γ, and thus different f t may prefer different γ for a trade-off. As in FIG0 (b) of the example, while choosing γ is to fix an x-coordinate section of margin distributions, its dual is to look for a y-section which leads to different margins for different f t. This motivates the quantile margin in the following theorem. Letγ q,f be the q th quantile margin of the network f with respect to sample S, DISPLAYFORM16 Theorem 2. Assume the input space is bounded by M > 0, that is x 2 ≤ M, ∀x ∈ X. Given a quantile q ∈, for any δ ∈ and τ > 0, the following holds with probability at least 1 − δ for all f t satisfyingγ q,ft > τ, DISPLAYFORM17 DISPLAYFORM18 Remark. We simply denote γ q,t for γ q,ft when there is no confusion. Compared with the bound, make the choice of γ varying with f t and the cost is an additional constant term C 2 q and the constraintγ q,t > τ that typically holds for large enough q in practice. In applications, stochastic gradient descent (SGD) often effectively improves the training margin distributions along the drops of training errors, a small enough τ and large enough q usually meetγ q,t > τ. Moreover, even with the choice τ = exp(−B), constant term [log log 2 (4(M + l)/τ )]/n = O(log B/n) is still negligible and thus very little cost is paid in the upper bound. In practice, tuning q ∈ is far easier than tuning γ > 0 directly and setting a large enough q ≥ 0.9 usually provides us lots of information about the generalization performance. The quantile margin works effectively when the dynamics of large margin distributions reflects the behavior of generalization error, e.g. FIG0. In this case, after certain epochs of training, the large margins have to be sacrificed to further improve small margins to reduce the training loss, that typically indicates a possible saturation or overfitting in test error. We briefly introduce the network and dataset used in the experiments. For the network, we first consider the convolutional neural network with very simple structure basic CNN(c). The structure is shown in Appendix Figure 7. Basically, it has five convolutional layers with c channels at each and one fully connected layer, where c will be specified in concrete examples. Second, we consider more practical network structure, AlexNet BID10, VGGNet-16 BID20 and ResNet-18 BID6. For the dataset, we consider CIFAR10, CIFAR100 BID9 ) and Mini-ImageNet.The spirit of the following experiments is to show, when and how, the margin bound could be used to predict the tendency of generalization or test error along the training path? This section is to apply Theorem 1 and Theorem 2 to predict the tendency of generalization error. Let's firstly consider training a basic CNN on CIFAR10 dataset with and without random noise. The relations between generalization error and training margin error e γ (f (x), y) with γ = 9.8, inverse quantile margin 1/γ q,t with q = 0.6 are shown in FIG1. In this simple example where the net is light and the dataset is simple, the linear bounds and show a good prediction power: they stop either near the epoch of sufficient training (Left, original data) or where even an overfitting occurs (Right, 10 percents label corrupted). and CIFAR10 with 10 percents label corrupted (Right). In each figure, we show training error (red solid), training margin error γ = 10 (red dash) and inverse quantile margin (red dotted) with q = 0.6 and generalization error (blue solid). The marker "x" in each curve indicates the global minimum along epoch 1,..., T. Both training margin error and inverse quantile margin successfully predict the tendency of generalization error. A few discussions are given below.1. There exists a trade-off on the choice of γ from the linear bounds (and parallel arguments hold for q). The training margin error with a small γ is close to the training error, while a large γ is close to generalization error and it's illustrated in Appendix Figure 8 where we show the Spearman's ρ rank correlation 1 between training margin error and training error, generalization error against threshold γ. 2. The training margin error (or inverse quantile margin) is closely related to the dynamics of training margin distributions. For certain choice of γ, if the curve of training margin error (with respect to epoch) is V-shape, the corresponding dynamics of training margin distributions will have a cross-over, where the low margins have a monotonic increase and the large margins undergo a phase transition from increase to decrease, as illustrated by the red arrow in FIG0. 3. Dynamics of quantile margins can adaptively select γ t for each f t without access to the complexity term. Unlike merely looking at the training margin error with a fixed γ, quantile margin bound shows a stronger prediction power than and even be able to capture more local information as illustrated in FIG2. The generalization error curve has two valleys corresponding to a local optimum and a global optimum, and the quantile margin curve with q = 0.95 successfully identifies both. However, if we consider the dynamics of training margin errors, it's rarely possible to recover the two valleys at the same time since their critical thresholds γ t1 and γ t2 are different. Another example of ResNet is given in Appendix Figure 9. In this section, we explore the normalized margin dynamics with over-parameterized models whose expression power might be greater than data complexity. We conduct experiments in the following two scenarios.1. In the first experiment shown in FIG3, we fix the dataset to be CIFAR10 with 10 percent of labels randomly permuted, and gradually increase the channels from basic CNN to basic CNN. As the channel number increases, dynamics of the normalized training margins in the first row change from a phase transition with a cross-over in large margins to a monotone improvement of margin distributions. This phenomenon is not a surprise since with a strong representation power, the whole training margin distribution can be monotonically improved without sacrificing the large margins. On the other hand, the generalization or test error can never be monotonically improved. In the second row, heatmaps depict rank correlations of dynamics between training and test margin errors, which clearly show the phase transitions for CNN and CNN and its disappearance for CNN. 2. In the second experiment shown in 5, we compare the normalized margin dynamics of training CNN and ResNet18 on two different datasets, CIFAR100 (the simpler) and Mini-ImageNet (the more complex). It shows that: (a) CNN (5.8M parameters) does not have an over-representation power on CIFAR100, whose normalized training margin dynamics exhibits a phase transition -a sacrifice of large margins to improve small margins during training; (b) ResNet18 (11M parameters) exhibits an over-representation power on CIFAR100 via a monotone improvement on training margins, but loses such a power in Mini-ImageNet with the phase transitions in margin dynamics. More experiments including AlexNet and VGG16 are shown in Appendix FIG0.This phenomenon is not unfamiliar to us, since Breiman BID1 has pointed out that the improvement of training margins is not enough to guarantee a small generalization or test error in the boosting type algorithms. In this paper Breiman designed an algorithm, called arc-gv, enjoying an uniformly better training margin distribution comparing with Adaboost but suffer a higher generalization error. Now again we find the same phenomenon ubiquitous in deep neural networks. Dataset: CIFAR100 (Left, Middle), Mini-ImageNet (Right) with 10 percent labels corrupted. With a fixed network structure, we further explore how the complexity of dataset influences the margin dynamics. Taking ResNet18 as an example, margin dynamics on CIFAR100 doesn't have any crossover (phase transition), but on Mini-Imagenet a cross-over occurs. In the end, it's worth mentioning different choices of the normalization factor estimates may affect the range of predictability. In all experiments above, normalization factor is estimated via an upper bound on spectral norm given in Appendix A (Lemma A.1 in Section A). One could also use power iteration BID14 to present a more precise estimation on spectral norm. It turns out a more accurate estimation of spectral norm can extend the range of predictability, but Breiman's dilemma is still there when the balance between model expression power and dataset complexity is broken. More experiments on this aspect can be found in FIG0 in Appendix. In this paper, we show that Breiman's dilemma is ubiquitous in deep learning, in addition to previous studies on Boosting algorithms. We exhibit that Breiman's dilemma is closely related to the tradeoff between model expression power and data complexity. A novel perspective on phase transitions in dynamics of Lipschitz-normalized margin distributions is proposed to inspect when the model has over-representation power compared to the dataset, instead of merely counting the number of parameters. A data-driven early stopping rule by monitoring the margin dynamics is a future direction to explore. Lipschitz semi-norm plays an important role in normalizing or regularizing neural networks, e.g. in GANs BID7 BID14, therefore a more careful treatment deserves further pursuits. In this section we discuss how to estimate the Lipschitz constant bound in. Given an operator W associated with a convolutional kernel w, i.e. W x = w * x, there are two ways to estimate its operator norm. We begin with a useful lemma, Lemma A.1. For convolution operator with kernel w, i.e. W x:= w * x, there holds w * x 2 ≤ w 1 x 2.In other words, W σ ≤ w 1.Proof. DISPLAYFORM0 where the second last step is due to Cauchy-Schwartz inequality. A. 1 -norm. The convolutional operator (spectral) norm can be upper bounded by the 1 -norm of its kernels, i.e. W σ ≤ w 1. This is a simple way but the bound gets loose when the channel numbers increase. B. Power iteration. A fast approximation for the spectral norm of the operator matrix is given in BID14 in GANs that is based on power iterations BID5 ). Yet as a shortcoming, it is not easy to apply to the ResNets. We compare two estimation in Appendix FIG0. It turns out both of them have prediction power on the tendency of generalization error and both of them will fail when the network has large enough expression power. Though using 1 norm of kernel is extremely efficient, the power iteration method may be tighter and has a wider range of predictability. In the remaining of this section, we will particularly discuss the treatment of ResNets. ResNet is usually a composition of the basic blocks shown in FIG5 with short-cut structure. The following method is used in this paper to estimate the upper bound of operator or spectral norm of such a basic block of ResNet. B are mean and variance of batch samples, while keeping an online averaging asμ andσ 2. Then BN rescales x + by estimated parametersα,β and outputx =αx + +β. Therefore the whole rescaling of BN on the kernel tensor w of the convolution layer isŵ = wα/ √σ 2 + and its corresponding rescaled operator is DISPLAYFORM1 (b) Activation and pooling: their Lipschitz constants L σ can be known a priori, e.g. L σ = 1 for ReLU and hence can be ignored. In general, L σ can not be ignored if they are in the shortcut as discussed below.(d) Shortcut: In residue net with basic block in FIG5, one has to treat the mainstream (Block 2, Block 3) and the shortcut Block 1 separately. Since f + g F ≤ f F + g F, in this paper we take the Lipschitz upper bound by DISPLAYFORM2 where Ŵ i σ denotes a spectral norm estimate of BN-rescaled convolutional operator W i. In particular L σout can be ignored since all paths are normalized by the same constant while L σin can not be ignored due to its asymmetry. B STRUCTURE OF BASIC CNN The picture is slight different here, since after the first (better) local minimum, the training margin distribution is uniformly improved without reducing generalization error. Therefore, we could not expect the inverse quantile margin to reflect the tendency of generalization error globally, especially the order of two local minimums. However, around epochs when local minimum occurs, the training margin distribution still has a cross-over, and thus the inverse quantile margin could reflect the tendency locally. Lemma D.1. For any δ ∈ and bounded-value functions F B:= {f : X → R : f ∞ ≤ B}, the following holds with probability at least 1 − δ, DISPLAYFORM3 where DISPLAYFORM4 is the Rademacher Complexity of function class F.For completeness, we include its proof that also needs the following well-known McDiarmid's inequality (see, e.g.). DISPLAYFORM5 where with probability at least 1 − δ, DISPLAYFORM6 by McDiarmid's bounded difference inequality, and DISPLAYFORM7 using Rademacher complexity. To see FORMULA0, we are going to show that sup f ∈F B E nf is a bounded difference function. Consider DISPLAYFORM8 Assume that the i-th argument x i changes to x i, then for every g, g(DISPLAYFORM9 Hence sup g g(x i, x −i) − sup g g(x i, x −i) ≤ B/n, which implies that sup f ∈F B E nf is a B/nbounded difference function. Then follows from the McDiarmid's inequality (Lemma D.2) using B i = B/n and δ = exp(−2nε 2 /B 2).As to, DISPLAYFORM10 that ends the proof. We also need the following contraction inequality of Rademacher Complexity BID11 BID13. Lemma D.3 (Rademacher Contraction Inequality). For any Lipschitz function: DISPLAYFORM11 has an additional factor 2 in the contraction inequality which is dropped in BID13. Its current form is stated in BID15 as Talagrand's Lemma (Lemma 4.2).Beyond, we further introduce the family, DISPLAYFORM12 and the sub-family constraint in Lipschitz semi-norm on f, DISPLAYFORM13 The following lemma BID8 allows us to bound the Rademacher complexity term of R n (G) by R n (H), DISPLAYFORM14 where the last inequality is implied from R n ({max(f 1, . . ., f M): BID8 BID15. DISPLAYFORM15 Proof of Proposition 1. Without loss of generality, we assume L σi = 1, i = 1,..., l. Let T (r) =: {t(x) = w · x: w 2 ≤ r} be the class of linear function with Lipschitz semi-norm less than r and we show that for each t ∈ T (L/2), there exists f ∈ F with f F ≤ L and y 0 ∈ {1, . . ., K} such that where f F ≤ Π l i=1 W i σ = 2L/2 = L, and thus h ∈ H L by definition. Therefore, R n (H L) ≥ R n (T (L/2)), DISPLAYFORM0 DISPLAYFORM1 where the second equality is implied from Cauchy-Schwarz inequality and the last inequality is implied from Khintchine inequality. Proof of Theorem 1. Consider l (γ1,γ2) (ζ(f (x), y)), wheref:= f /L f is the normalized network, ζ(f (x), y) ∈ G 1. Then for any γ 2 > γ 1 ≥ 0, DISPLAYFORM0 ≤ P n (γ1,γ2) (f (x), y) + 2R n (l (γ1,γ2) • G 1 ) + log(1/δ) 2n, ≤ P n (γ1,γ2) (f (x), y) + 2 ∆ R n (G 1) + log(1/δ) 2n, ≤ P n γ1,γ2 (f (x), y) + 2K 2 ∆ R n (H 1) + log(1/δ) 2n, ≤ P n γ2 (f (x), y) + 2K 2 ∆ R n (H 1) + log(1/δ) 2n, where the first and last inequality is implied from 1[ζ < γ 1] ≤ (γ1,γ2) (ζ) ≤ 1[ζ < γ 2], the second inequality is a direct consequence of Lemma D.1, the third inequality from Rademacher Contraction Inequality (Lemma D.3) and finally the fourth equation is implied from Lemma D.4. Proof of Theorem 2. Firstly, we show after normalization, the normalize margin has an upper bound, DISPLAYFORM0, where x i = σ i (W i x i−1 + b i) with x 0 = x,W i = (W i, b i) and L σi is the Lipschitz constant of activation function σ i with σ i = 0, i = 1,..., L. Then, for normalized networkf = f /L f with DISPLAYFORM1 Therefore ζ(f (x), y) ≤ 2 f (x) 2 = 2(M + L) =: M 1, and the quantile margin is also bounded γ q,t ≤ M 1 for all q ∈, t = 1,..., T.The remaining proof is standard. For any > 0, we take a sequence of k and γ k, k = 1, 2,... by k = + log k n and γ k = M 1 2 −k. Then by Theorem 1, DISPLAYFORM2 where A k is the event P[ζ(f t (x), y) < 0] > P n [ζ(f (x), y) < γ k ] + 2K 2 γ k R(H 1) + k, and the probability is taken over samples {x 1, ...x n}. We further consider the probability for none of A k occurs, DISPLAYFORM3 2 ), ≤ 2 exp(−2n 2).Hence, fix a q ∈, for any t = 1,..., T, as long asγ q,t > 0, there exists ak ≥ 1 such that, γk +1 ≤γ q,t < γk. Therefore, DISPLAYFORM4 ⊇ P[ζ(f t (x), y) < 0] > P n [ζ(f t (x), y) <γ q,t ] + 4K 2 γ q,t R(H 1) + k +1, = P[ζ(f t (x), y) < 0] > P n [ζ(f t (x), y) >γ q,t ] + 4K 2 γ q,t R(H 1) + + log(k + 1) n, ⊇ P[ζ(f t (x), y) < 0] > P n [ζ(f t (x), y) >γ q,t ] + 4K 2 γ q,t R(H 1) + + log log 2 (2M 1 /γ q,t) n.The first inequality is implied from P n [ζ(f t (x), y) <γ q,t ] > P n [ζ(f t (x), y) < γk +1 ], since γk +1 ≤ γ q,t. The second inequality is implied fromγ q,t < 2γk +1 and thus, 1/γk +1 < 2/γ q,t. The third equality is the direct definition of k. The last inequality is implied fromk + 1 = log 2 (M 1 /γk +1) and again, 1/γk +1 < 2/γ q,t. The is proved immediately if we do a transform from to δ.

Represent the TLDR for finding the corresponding scientific article

Bregman's dilemma is shown in deep learning that improvement of margins of over-parameterized models may result in overfitting, and dynamics of normalized margin distributions are proposed to predict generalization error and identify such a dilemma.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: It has been an open research challenge for developing an end-to-end multi-domain task-oriented dialogue system, in which a human can converse with the dialogue agent to complete tasks in more than one domain. First, tracking belief states of multi-domain dialogues is difficult as the dialogue agent must obtain the complete belief states from all relevant domains, each of which can have shared slots common among domains as well as unique slots specifically for the domain only. Second, the dialogue agent must also process various types of information, including contextual information from dialogue context, decoded dialogue states of current dialogue turn, and queried from a knowledge base, to semantically shape context-aware and task-specific responses to human. To address these challenges, we propose an end-to-end neural architecture for task-oriented dialogues in multiple domains. We propose a novel Multi-level Neural Belief Tracker which tracks the dialogue belief states by learning signals at both slot and domain level independently. The representations are combined in a Late Fusion approach to form joint feature vectors of (domain, slot) pairs. Following recent work in end-to-end dialogue systems, we incorporate the belief tracker with generation components to address end-to-end dialogue tasks. We achieve state-of-the-art performance on the MultiWOZ2.1 benchmark with 50.91% joint goal accuracy and competitive measures in task-completion and response generation. In a task-oriented dialogue system, the Dialogue State Tracking (DST) module is responsible for updating dialogue states (essentially, what the user wants) at each dialogue turn. The DST supports the dialogue agent to steer the conversation towards task completion. As defined by Henderson et al. (2014a), a dialogue belief state consists of inform slots -information to query a given knowledge base or database (DB), and request slots -information to be returned to the users. Task-oriented dialogues can be categorized as either single-domain or multi-domain dialogues. In single-domain dialogues, humans converse with the dialogue agent to complete tasks of one domain. In contrast, in multi-domain dialogues, the tasks of interest can come from different domains. A dialogue state in a multi-domain dialogue should include all inform and request slots of corresponding domains up to the current turn. Examples of a single-domain dialogue and a multi-domain dialogue with annotated states after each turn can be seen in Figure 1. Despite there being several efforts in developing task-oriented dialogue systems in a single domain (a;), there have been limited contributions for multi-domain task-oriented dialogues. Developing end-to-end systems for multi-domain dialogues faces several challenges: Belief states in multi-domain dialogues are usually larger and more complex than in single-domain, because of the diverse information from multiple domains. Each domain can have shared slots that are common among domains or unique slots that are not shared with any. In an end-to-end system, the dialogue agent must incorporate information from source sequences, e.g. dialogue context and human utterances, as well as tracked belief states and extracted information from knowledge base, to semantically shape a relevant response with accurate information for task completion. Directly applying methods for single-domain dialogues to multi-domain dialogues is not straightforward because the belief states extend across multiple domains. A possible solution is to process a multi-domain dialogue for N D times for N D domains, each time obtaining a belief state of one domain. However, this approach does not allow learning co-references in dialogues whereby users can switch from one domain to another turn by turn. We propose an end-to-end dialogue system approach which explicitly track the dialogue states in multiple domains altogether. Specifically, we propose Multi-level Neural Belief Tracker to process contextual information for both slot-level and domain-level signals independently. The two levels are subsequently combined to learn multi-domain dialogue states. Our dialogue state tracker enables shared learning of slots common among domains as well as learning of unique slots in each domain. we utilize multi-head attention layers to comprehensively process various types of information: dialogue context, user utterances, belief states of both inform and request slots, and DB query . The multi-head structure allows the model to independently attend to the features over multiple representation sub-spaces; and we combine all components to create a dialogue system from state tracking to response generation. The system can be jointly learned in an end-to-end manner. Our end-to-end dialogue system utilizes supervision signals of dialogue states and output responses without using system action annotation. To comprehensively validate our method, we compare our models with baselines in end-to-end, DST, and context-to-text generation settings. We achieve the state-of-the-art performance in DST, task-completion, and response generation in the MultiWOZ2.1 corpus ) as compared to other baselines in similar settings. In context-to-text generation setting that allows supervision of dialogue acts, our models can achieve competitive measures of Inform and BLEU metric. Our work is related to 2 main bodies of research: DST and end-to-end dialogue systems. Prior DST work focuses on single-domain dialogues using WOZ and DSTC2 (a) corpus. (Mrkšić et al., 2015; b;) address transfer learning in dialogues from one domain to another rather than multiple domains in a single dialogue. Our work is more related to recent effort for multi-domain DST such as; a;. These models can be categorized into two main categories of DST: fixed-vocabulary and open-vocabulary approach. Fixed vocabulary models assume known slot ontology with a fixed candidate set for each slot. Open-vocabulary models (; a;) derive the candidate set based on the source sequence i.e. dialogue history, itself. Our approach is more related to open-vocabulary approach as we aim to dynamically generate dialogue state based on input dialogue history. Different from most prior work, our Multi-level Neural Belief Tracker can learn domain-level and slot-level signals independently and both are combined in a Late Fusion manner to obtain contextual representations of all (domain, slot) pairs. Conventionally, an end-to-end dialogue system is composed of separate modules for Natural Language Understanding (NLU) , DST (b;), Dialogue Policy (;, and Natural Language Generator (NLG) (a;). These components can be learned independently and combined into a pipeline architecture for end-to-end system (; ; . Another line of research aims to develop a dialogue agent without modularizing these components but incorporating them into a single network (; ; ; b). Our work is more related to the latter approach whereby we incorporate conventional components into an integrate network architecture and jointly train all parameters. However, following , we consider a separate module that combines NLU and DST together. The module utilizes additional supervision for more fine-grained tracking of user goals. This strategy is also suitable for large-scale knowledge base with large number of entities. (; b;) completely omit the DST component by formulating entity attributes into memory form based on (Subject, Relation, Object) tuples. These models achieve good performance in small-scale corpus such as In-Car Assistant and WOZ2.0 but will become extremely hard to scale to large knowledge base in multi-domain setting such as MultiWOZ corpus. Given a dialogue with dialogue history of t − 1 turns, each including a pair of user utterance and system response, (U 1, S 1),..., (U t−1, S t−1), the user utterance at current dialogue turn U t, and a knowledge base in form of entity data tables, the goal of a task-oriented dialogue system is to generate a response S t that is not only appropriate to the dialogue context, but also task-related with the correct entity for the users. In the multi-domain dialogue setting, turns in the dialogue history and the current user utterance could come from different domains. Therefore, the generated response in this setting should also be domain-related with the correct domain-specific information for the users. We propose a novel Multi-level Neural Belief Tracker to track belief states at both domain level and slot level to address multi-domain dialogues. Following , we utilize the previous belief states B t−1 as an input to the model. This allows the model to rely on the dialogue states detected from the previous dialogue step t − 1 to update the state of the current step t. In addition, we adopt the attention-based principle of Transformer network and propose an end-to-end architecture for task-oriented dialogues. Our model allows comprehensive information processing from different input sources, incorporating contextual features from dialogue context and user utterance as well as learning signals from domain-level and slot-level dialogue states. Our solution consists of 3 major components: (i) Encoders encode sequences of dialogue history, current user utterances, target system responses, domain and slot names, and previous dialogue belief states, into continuous representations. (ii) Multi-level Neural Belief Tracker includes 2 modules, one for processing slot-level information and one for domain-level information. Each module comprises attention layers to project domain or slot token representations and attend on relevant features for state tracking. The outputs of the two modules are combined to create domain-slot joint feature representations. Each feature representation is used as a context-aware vector to decode the corresponding inform or request slots in each domain. (iii) Response Generator projects the target system responses and incorporates contextual information from dialogue context as well as intermediate variables from the state tracker and DB query . Employing attention mechanisms with feed-forward and residual connections allows our models to focus on relevant parts of the inputs and pass on the relevant information to decode appropriate system responses. We combine all the modules into an end-to-end architecture and jointly train all components. An overview of the proposed approach can be seen in Figure 2. An encoder encodes a text sequence of tokens (x 1, ..., x n) to a sequence of continuous representation z = (z 1, ..., z n) ∈ R n×d. Each encoder includes a token-level trainable embedding layer and layer normalization . Depending on the type of text sequences, we inject sequential characteristics of the tokens (i.e. their positions in the sequence) using a sine and cosine positional encoding functions . Element-wise summation is used to combine the token- level embedding with positional embedding, each has the same embedding dimension d. The current user utterance U t is tokenized, prepended and appended with sos and eos token respectively. In the dialogue history, each human utterance and system response up to dialogue step t − 1 is processed similarly. The tokenized past utterances and system responses are concatenated sequentially by the dialogue step. For target system response S t, during training, the sequence is offset by one position to ensure that token prediction in generation step i is based on the previous positions only i.e. 1,..., i − 1. Denoting name sloti and value sloti as the slot name and slot value of slot i, we create sequences of dialogue belief state from previous turn by following the template: value sloti inf _name sloti... req_name slotj... domain d... A req_name slotj is only included in the sequence if slot j is predicted as in the previous turn. As a slot such as area can be both request or inform type, the 2 slot types are differentiated by the prefixes inf and req. Our belief sequences can be used to encode past dialogue states of multiple domains, each separated by the domain d token. To learn slot-level and domain-level signals for state tracking, we construct set of slot and domain tokens as input to the state tracker. Each input set is created by concatenating slot names or domains: respectively. Both sequences are kept fixed in all dialogue samples to factor in all possible domains and slots for multi-domain state tracking. Positional encoding is used in all sequences except for input sets of slot and domain tokens as these sets do not contain sequential characteristic. Embedding weights are shared among all the encoders of source sequences. Embedding weights of the target system responses are not shared to allow the models to learn the semantics of input and output sequences differently. The DST module processes slot-level and domain-level information independently, and integrates the two for multi-domain state tracking. We adopt a Late Fusion approach to combine domain and slot representations in deeper network layers. Slot-level Processing. Given the encoded features from the source sequences, including dialogue history z his, previous belief state z bs, and the current user utterance z utt, the slot-level signals are learned by projecting the encoded slot token sequence z S through N S dst identical layers. Each layer contains 4 attention blocks, each of which employ the multi-head attention mechanism to attend on the inputs at different representation sub-spaces. Each attention block is coupled with a position-wise feed-forward layer, including 2 linear transformations with ReLU activation in between. Residual connection and layer normalization are employed in each attention block. Specifically, given the current feature vector z out S as output from previous attention block (or z S itself in the first attention block of the first processing layer) and the encoded features z seq of a source sequence, the multi-head attention is defined as: where, and seq = {S, his, bs, utt} (for simplicity, the subscripts of S and seq are omitted in each W). The first attention block is a self-attention, i.e. seq = S, which enables learning the relation between slots independently from domains. Subsequent attention layers on dialogue context, previous belief state, and user utterance of current turn, inject each slot token representation with dialogue contextual information up to current user utterance in turn t. Through residual connection, the contextual information are passed forward in each z out S. Using different attention blocks allows flexible processing of information from various input sources. Domain-level Processing. The input to the domain-level processing module includes the encoded domain token sequence z D, the encoded dialogue history up to turn t − 1 z his, and the encoded user utterance of current turn z utt. The domain features are passed through N D dst identical layers, each of which include 3 multi-head attention blocks to obtain important contextual information from dialogue context and user utterance. Similarly to slot-level processing, a self-attention block is leveraged to allow reasoning among domains independently from slots. Attending on dialogue history and current user utterance separately enables learning domain signals from the contextual information of past dialogue turns and current turns differently. Therefore, the models can potentially detect changes of dialogue domains from past turns to the current turn. Especially in multi-domain dialogues, users can switch from one domain to another and the generated responses should address the latest domain. d is used to decode the corresponding domain-specific slot i. The vector is used as initial hidden state for an RNN decoder to decode an inform slot token by token or passed through a linear transformation layer for binary classification for a request slot. The decoded dialogue states are used to query the DBs of all domains and obtain the number of the entities in each domain. We then create a fixed-dimensional one-hot pointer vector for each domain d: z We embed the pointer vector with the learned embedding and positional embedding as similarly described in Section 3.1, ing in z db ∈ R 6N D ×d. The DB pointer vector z db, context-aware domain-slot joint features z out DS, encoded dialogue history z his, and user utterance of current turn z utt, are used as inputs to incorporate relevant signals to decode system responses. The generator includes N gen identical layers, each includes 5 multi-head attention blocks, including a self-attention block at the beginning. Adopting attention with residual connection in each block allows the models to comprehensively obtain contextual cues, either through text sequences or domain-slot joint features, and knowledge base signals from DB pointer vectors. The final output z out gen is passed to a linear transformation with softmax activation to decode system responses. The objective function is a combination of belief state objectives, including the log-likelihood of all inform slot sequences S inf, and the binary cross entropy of request slots S req, and the system response objective, including the log-likelihood of the target response sequence T, as follows: where The above objectives are conditioned on the input features, including dialogue context C, current user utterance U, previous and current belief state B t−1 and B t, and DB queries Q. 4.1 DATA We used the MultiWOZ 2.1 dataset ) which consists of both single-domain and multi-domain dialogues. Compared to version 2.0, MultiWOZ 2.1 is improved with some correction of DST labels, including about 40% changes across training samples. We pre-processed the dialogues by tokenizing, lower-casing, and delexicalizing all system responses. From the belief state annotation of the training data, we identified all possible domains and slots. We identified N D = 7 domains and N S = 35 unique inform slots in total. We followed the preprocessing scripts as provided by b). The corpus includes 8,438 dialogues in the training with an average of 1.8 domains per dialogue. Each dialogue has more than 13 turns. There are 1,000 in each validation and test set, each including an average of 1.9 domains per dialogue. Other details of data pre-processing procedures, domains, slots, and entity DBs, are included in Appendix A.1. The model parameters are: We employed dropout of 0.3 at all network layers except the linear layers in the generative components. Label smoothing for target system responses is applied during training. During training, we utilize teacher-forcing learning strategy by simply using the ground-truth inputs of dialogue state from previous turn and the gold DB pointer. During inference, in each dialogue, we decode system responses sequentially turn by turn, using the previously decoded belief state as input in the current turn, and at each turn, using the decoded belief state to query DBs for pointer vectors. We train all networks in an end-to-end manner with Adam optimizer and the learning rate schedule similarly adopted by. We used batch size 32 and tuned the warmup_steps from 9K to 15K training steps. All models are trained up to 30 epochs and best models are selected based on validation loss. We used a greedy approach to decode all slots and beam search with beam size 5 and a length penalty 1.0 to decode responses. The maximum length is set to 10 tokens for each slot and 20 for system responses. Our models are implemented using PyTorch . To evaluate the models, we use the following metrics: DST metrics: Joint Accuracy and Slot Accuracy (b). Joint Accuracy compares the predicted dialogue states to the ground truth in each dialogue turn. All slot values must match the ground truth labels to be counted as a correct prediction. Slot Accuracy considers individual slot-level accuracy across the topology. Task-completion metrics: Inform and Success . Inform refers to system ability to provide an appropriate entity while Success is the system ability to answer all requested attributes. Generation metrics: BLEU score . We ran all experiments 3 times and reported the average . We report in 2 different settings: end-to-end dialogues and DST. In end-to-end setting, we train a dialogue agent that is responsible for both DST and text generation without assuming access to ground-truth labels. End-to-End. In this setting, we compare our model performance on the joint task of DST and context-to-text generation. For fair comparision, we select TSCP as the baseline as TSCP does not use additional supervision signals of system action as input. This is the current state-of-the-art for end-to-end dialogue task in the single-domain WOZ . TSCP applies pointer network to develop a two-stage decoding process to decode belief states, in a form of text sequence, and subsequently decode system responses. We adapt the method to the multi-domain dialogue setting. We experiment with 2 cases of TSCP in which the maximum length of the output state sequence L bspan is set to 8 and 20 tokens. As can be seen in Table 1, our model outperforms in all metrics, except for the Slot Acc metric in one case. Overall, our model performs well in both multi-domain and single-domain dialogues, especially with higher performance gain in multi-domain dialogues. The performance gain in multi-domain dialogues can be explained by the separate network structure between domain and slot processing modules in our models. This allows our models to learn domain-dependent and slot-dependent signals separately before the two are fused into a joint feature vectors for downstream process. For TSCP, increasing the L bspan from 8 to 20 tokens helps to improve the performance, but also increases the training time to convergence significantly. In our approach, all inform and request slots are decoded independently and the training time is less affected by the size of the target dialogue states, especially in cases of extensive belief states (e.g. 4 or 5 domains in a dialogue). Additional by individual domains are described in Appendix A.3. DST. We isolate the DST components (i.e. training models only with L(B t)) and report the DST performance. We compare the performance with the baseline models on the MultiWOZ 2.1 in Table 2 (Refer to Appendix A.2 for more description of DST baselines). Our model outperforms existing baselines and achieves the state-of-the-art performance in MultiWOZ2.1 corpus. By leveraging on dialogue context signals through independent attention modules at domain level and slot level, our DST can generate slot values more accurately. DST approaches that try to separate domain and slot signals such as TRADE (a) reveal competitive performance. However, our approach has better performance as we enable deeper interaction of context-related signals in each domain and slot representation. Compared to TRADE, our approach can be considered as Late Fusion approach that combines representations in deeper network layers for better joint features of domains and slots. We also noted that DST performance improves when our models are trained as an end-to-end system. This can be explained as additional supervision from system responses not only contributes to learn a good response generation network but also positively impact DST network. Additional DST of individual domains can be seen in Appendix A.3. For completion, we also conduct experiment for context-to-text generation setting and compare with baseline models in Appendix A.3. We experiment with different model variants in Table 3. First, we noted that removing selfattention on the joint feature domain-slot vectors (N DS dst = 0) reduces the joint accuracy performance. This self-attention is important because it allows our models to learn signals across (domain, slot) joint features rather than just at independently domain level and slot level. Second, ranging the number of attention layers in domain-level processing and slot-level processing from 3 to 1 gradually reduces the model performance. This shows the efficacy of our Late Fusion approach. Combining the Joint Accuracy HJST 35.55% DST Reader 36.40% TSCP 37.12% FJST 38.00% HyST 38.10% TRADE (a) 45.60% Ours 49.55% features at deeper network layers in better joint feature representation and hence, increases the model performance. Lastly, we observed that our models can efficiently detect contextual signals from the dialogue states of previous turn PrevBS as the performance of our models with or without using the full dialogue history is very similar. This will benefit as the dialogue history evolves over time and our models only need to process the latest dialogue turn in combination with the predicted dialogue state in previous turn as an input. Qualitative Analysis. We examine an example dialogue in the test data and compare our predicted outputs with the baseline TSCP (L bspan = 20) and the ground truth. From the table in the left of Figure 3, we observe that both our predicted dialogue state and system response are more correct than the baseline. Specifically, our dialogue state can detect the correct type slot in the attraction domain. As our dialogue state is correctly predicted, the queried from DB is also more correct, ing in better response with the right information (i.e. 'no attraction available'). From visualization of domain-level and slot-level attention on the user utterance, we notice important tokens of the text sequences, i.e.'entertainment' and'close to', are attended with higher attention scores. In addition, at domain-level attention, we find a potential additional signal from the token'restaurant', which is also the domain from the previous dialogue turn. We also observe that attention is more refined along the neural network layers. For example, in the domain-level processing, compared to the 2 nd layer, the 4 th layer attention is more clustered around specific tokens of the user utterance. The complete predicted output for this example dialogue and other qualitative analysis can be seen in Appendix A.4. In this work, we proposed an end-to-end dialogue system with a novel Multi-level Neural Belief Tracker. Our DST module can track complex belief states of multiple domains and output more accurate dialogue states. The DST is combined with attention-based generation module to generate dialogue responses. Evaluated on the large-scale multi-domain dialogue benchmark MultiWOZ2.1, our models achieve the state-of-the-art performance in DST and competitive measures in taskcompletion and response generation. Figure 3: Example dialogue with the input system response St−1 and current user utterance Ut, and the output belief state BSt and system response St. Compared with TSCP (Row 3), our dialogue state and response (Last Row) are more correct and closer to the ground truth (Row 2). Visualization of attention to the user utterance sequence at slot-level (lower right) and domain-level (upper right) is also included. More red denotes higher attention score between domain or slot representation and token representation. Best viewed in color. A.1 DATA PRE-PROCESSING First, we delexicalize each target system response sequence by replacing matched entity attribute appeared in the sequence to the canonical tag domain_attribute. For example, the original target response'the train id is tr8259 departing from cambridge' is delexicalized into'the train id is train_id departing from train_departure'. We use the provided entity databases (DBs) to match potential attributes in all target system responses. For dialogue history, we keep the original version of all text, including system responses of previous turns, rather than the delexicalized form. We split all sequences of dialogue history, user utterances of current turn, previous belief states, and delexicalized target responses, into case-insensitive tokens. We share the embedding weights of all source sequences. For source sequences, in total there are 5,491 unique tokens, including slot and domain tokens as well as eos, sos, pad, and unk tokens. For target sequences, there are 2,648 unique tokens in total, including all canonical tags as well as eos, sos, pad, and unk tokens. As can be seen in Table 4, in source sequences, the overlapping rates of unique tokens to the training embedding vocabulary are about 64% and 65% in validation and test set respectively. For target sequences, the overlapping rates are about 83% and 82% in validation and test set respectively. As we analyze the data, we summarize the number of dialogues in each domain in Table 5. For each domain, a dialogue is selected as long as the whole dialogue (i.e. single-domain dialogue) or parts of the dialogue (i.e. in multi-domain dialogue) is involved with the domain. For each domain, we also build a set of possible inform and request slots using the belief state annotation in the training data. The details of slots, entity attributes, and DB size, in each domain, can be seen in Table 6. The DBs of 3 domains taxi, police, and hospital were not provided in the benchmark. We describe a list of baseline models in DST setting and context-to-text generation setting. FJST and HJST . FJST and HJST follow a fixed-vocabulary approach for state tracking. Both models include encoder modules (either bidirectional LSTM or hierarchical LSTM) to encode the dialogue history. The models pass the context hidden states to separate linear transforma- tion to obtain final vectors to predict individual state slots separately. The output vector is used to measure a score of a predefined candidate set for each slot. TSCP . TSCP is an end-to-end dialogue system that can do both DST and NLG. The model utilize pointer network to generate both dialogue states and responses. To compare with TSCP in DST setting, we adapt the model to multi-domain dialogues and report the only on DST components. For DST experiment, we reported the performance when the maximum length of dialogue state sequence in the state decoder L is set to 20 tokens. DST Reader . This model considers the DST task as a reading comprehension task. The model predicts each slot as a span over tokens within dialogue history. DST Reader utilizes attention-based neural network with additional modules to predict slot type and slot carryover probability. HyST. This baseline combines the advantage of both fixed-vocabulary and open-vocabulary approaches. In open-vocabulary, the set of candidates of each slot is constructed based on all word n-grams in dialogue history. Both approaches are applied in all slots and depending on their performance in validation set, the better approach is applied to predict individual slots. TRADE (a). This is the current state-of-the-art model on the MultiWOZ2.1 dataset. The model combines pointer network to generate individual slot token-by-token. The prediction is additional supported by a slot gating component that decides whether the slot is "none", "dontcare", or "pointer" (generated). provides a baseline for this setting by following the sequence-to-sequence model with additional signals from the belief tracker and discrete data pointer vector. TokenMoE . TokenMoE refers to Token-level Mixture-of-Expert model. The model follows a modularized approach by separating different components known as expert bots for different dialogue scenarios. A dialogue scenario can be dependent on a domain, a type of dialogue act, etc. A chair bot is responsible for controlling expert bots to dynamically generate dialogue responses. HDSA . This is the current state-of-the-art for context-to-text generation setting in MultiWOZ2.0. HDSA leverages the structure of dialogue acts to build a multi-layer hierarhical graph. The graph is incorporated as an inductive bias in self-attention network to improve the semantic quality of generated dialogue responses. Structured Fusion . This approach follows a traditional modularized dialogue system architecture, including separate components for NLU, DM, and NLG. These components are pretrained and combined into an end-to-end system. Each component output is used as a structured input to other components. LaRL . This model uses a latent dialogue action framework instead of traditional handcrafted dialogue acts. The latent variables are learned using unsupervised learning with stochastic variational inference. The model are trained in a reinforcement learning framework whereby the parameters are trained to yield better Success rate. Domain-Specific Results. In Table 7 and 8, we presented additional of our model and the baselines TSCP . For state tracking, the metrics are calculated for domain-specific slots of the corresponding domain at each dialogue turn. For task completion and response generation, we calculated the metrics for single-domain dialogues of the corresponding domain. We do not report the Inform metric for the taxi domain because no DB was provided in the benchmark for this domain. From Table 7, in each domain, our approach outperforms TSCP across most of the metrics, except the Success and BLEU metric in the taxi domain. In term of task-completion, our model performs better significant improvement in domains with large DB sizes such as train and restaurant. In term of response generation, our are consistently higher than the baselines as the model can return more appropriate responses from better decoded dialogue states and DB queried . For state tracking task alone, in Table 8, our models perform consistently in the 3 domains attraction, restaurant, and train domains. However, the performance significantly drops in the taxi domain. This performance drop negatively impacts the overall performance across all domains. We plan to investigate further to identify and address challenges in this particular domain in future work. Context-to-Text Generation. , to compare with baselines in this setting, we assumes access to the ground-truth labels of dialogue belief states and data pointer during inference. We compare with existing baselines in Table 2 (Refer to Appendix A.2 for more description of the baselines). Our model achieves the state-of-the-art in the Inform metric but do not perform as well in terms of Success metric. We achieve a competitive BLEU score, only behind the current state-of-the-art HDSA model. An explanation for our model not able to achieve a high Success metric is that we did not utilize the dialogue act information. The current state-of-the-art HDSA leverages the graph structure of dialogue acts into dialogue models. Furthermore, compared to approaches such Table 9: Performance for context-to-text generation setting on MultiWOZ2.0. The baseline are as reported in the benchmark leaderboard. Inform Success BLEU Baseline 71.29% 60.96% 18.80 TokenMoE 75.30% 59.70% 16.81 HDSA 82.90% 68.90% 23.60 Structured Fusion 82.70% 72.10% 16.34 LaRL 82.78% 79.20% 12.80 Ours 83.83% 67.36% 19.88 as , our model does not use pretrained network modules such as NLU and DST. Our end-to-end setting is more related along the line of research work for end-to-end dialogue systems without relying on system action annotation (; ; b). To improve the Success metric, we plan to extend our work in the future that can derive better dialogue policy for higher Success rate. In Table 10, we reported the complete output of an example multi-domain dialogue. Overall, our dialogue agent can carry a proper dialogue with the user throughout the dialogue steps. Specifically, we observed that our model can detect new domains at dialogue steps where the domains are introduced e.g. attraction domain at the 5 th turn and taxi domain at the 8 th turn. The dialogue agent can also detect some of the co-references among the domains. For example, at the 5 th turn, the dialogue agent can infer the slot area for the new domain attraction as the user mentioned'close the restaurant'. We noticed that that at later dialogue steps such as the 6 th turn, our decoded dialogue state is not correct possibly due to the incorrect decoded dialogue state in the previous turn, i.e. 5 th turn. In Figure 4 and 5, we reported the Joint Goal Accuracy and BLEU metrics of our model by dialogue turn. As we expected, the Joint Accuracy metric tends to decrease as the dialogue history extends over time. The dialogue agent achieves the highest accuracy in state tracking at the 1 st turn and gradually reduces to zero accuracy at later dialogue steps, i.e. 15 th to 18 th turns. For response generation performance, the trend of BLEU score is less obvious. The dialogue agent obtains the highest BLEU scores at the 3 rd turn and fluctuates between the 2 nd and 13 th turn.

Represent the short summary for finding the corresponding scientific article (long)

We proposed an end-to-end dialogue system with a novel multi-level dialogue state tracker and achieved consistent performance on MultiWOZ2.1 in state tracking, task completion, and response generation performance.

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.