Question: Represent the text to find a very short summary (1-2 sentences max)

Question: To provide principled ways of designing proper Deep Neural Network (DNN) models, it is essential to understand the loss surface of DNNs under realistic assumptions. We introduce interesting aspects for understanding the local minima and overall structure of the loss surface. The parameter domain of the loss surface can be decomposed into regions in which activation values (zero or one for rectified linear units) are consistent. We found that, in each region, the loss surface have properties similar to that of linear neural networks where every local minimum is a global minimum. This means that every differentiable local minimum is the global minimum of the corresponding region. We prove that for a neural network with one hidden layer using rectified linear units under realistic assumptions. There are poor regions that lead to poor local minima, and we explain why such regions exist even in the overparameterized DNNs. Deep Neural Networks (DNNs) have achieved state-of-the-art performances in computer vision, natural language processing, and other areas of machine learning. One of the most promising features of DNNs is its significant expressive power. The expressiveness of DNNs even surpass shallow networks as a network with few layers need exponential number of nodes to have similar expressive power . The DNNs are getting even deeper after the vanishing gradient problem has been solved by using rectified linear units (ReLUs) BID12. Nowadays, RELU has become the most popular activation function for hidden layers. Leveraging this kind of activation functions, depth of DNNs has increased to more than 100 layers BID7.Another problem of training DNNs is that parameters can encounter pathological curvatures of the loss surfaces prolonging training time. Some of the pathological curvatures such as narrow valleys would cause unnecessary vibrations. To avoid these obstacles, various optimization methods were introduced (; BID9 . These methods utilize the first and second order moments of the gradients to preserve the historical trends. The gradient descent methods also have a problem of getting stuck in a poor local minimum. The poor local minima do exist in DNNs, but recent works showed that errors at the local minima are as low as that of global minima with high probability BID4 BID2 BID8 BID14 ).In case of linear DNNs in which activation function does not exist, every local minimum is a global minimum and other critical points are saddle points BID8. Although these beneficial properties do not hold in general DNNs, we conjecture that it holds in each region of parameters where the activation values for each data point are the same as shown in FIG0. We prove this for a simple network. The activation values of a node can be different between data points as shown in FIG0, so it is hard to apply proof techniques used for linear DNNs. The whole parameter space is a disjoint union of these regions, so we call it loss surface decomposition. Using the concepts of loss surface decomposition, we explain why poor local minima do exist even in large networks. There are poor local minima where gradient flow disappears when using the ReLU . We introduce another kind of poor local minima where the loss is same as that of linear regression. To be more general, we prove that for each local minimum in a network, there exists a local minimum of the same loss in the larger network that is constructed by adding a node to that network. DISPLAYFORM0 T. In each region, activation values are the same. There are six nonempty regions. The parameters on the boundaries hit the non-differentiable point of the rectified linear unit. Loss surface of deep linear networks have the following interesting properties: 1) the function is non-convex and non-concave, 2) every local minimum is a global minimum, 3) every critical point that is not a global minimum is a saddle point BID8. This means that there is no poor local minima problem when using gradient descent methods, but such properties do not hold for nonlinear networks. We conjecture that these properties hold if activation values are fixed, and we prove it for a simple network. The loss surface of DNNs can be decomposed into regions in terms of activation values as illustrated in FIG0. Let D be a dataset {(x 1, y 1), (x 2, y 2),..., (x N, y N)} with x i ∈ R n and y i ∈ R. We define a network with one hidden layer as follows: DISPLAYFORM0 The model parameters are W ∈ R h×n, v ∈ R h, b ∈ R h, and c ∈ R where h is the number of hidden nodes. Let θ = [vec(W), v, b, c] T collectively denote vectorized form of all the model parameters. The activation function σ(x) = max(x, 0) is a rectified linear unit, and we abuse notation by generalizing it as an element-wise function for multidimensional inputs. Alternatively, the network can be expressed in terms of the activation values: DISPLAYFORM1 where DISPLAYFORM2 T is a vector of the binary activation values a ij ∈ {0, 1} of i-th data point x i, and A = (a 1, a 2, ..., a N) is a collection of all activation values for a given dataset D. We fix the activation values of the function g A (x i, θ) regardless of real activation values to find out the interesting properties. The real model f (x i, θ) agrees with g A (x i, θ) only if A is same as the real activation values in the model. Before we introduce a definition of the activation region, we denote w A simple example of a non-differentiable local minimum for a dataset x 1 = −2, y 1 = −3, x 2 = +2, y 2 = −1. In this example, a network is defined by f (x) = w 2 σ(w 1 x) and w 1 is fixed to one. The non-differentiable local minima exist in a line w 1 = 0 which is a boundary of the two regions. Note that if DISPLAYFORM3 We consider a general loss function called squared error loss: DISPLAYFORM4 The following lemma state that the local curvatures of DISPLAYFORM5 Lemma 2.2 For any differentiable point θ ∈ R A, the θ is a local minimum (saddle point) in L f (θ) if and only if it is a local minimum (saddle point) in L g A (θ). The function g A (x i, θ) of fixed activation values A has properties similar to that of linear neural networks. If all activation values are one, then the function g A (x i, θ) is identical to a linear neural network. In other cases, some of the parameters are inactive. The proof becomes tricky since inactive parameters are different for each data point. In case of the simple network g A (x i, θ), we can convert it into a convex function in terms of other variables. DISPLAYFORM0 where p j = v j w j and q j = v j b j. The v j is a j-th scalar value of the vector v and a ij is an activation value on a j-th hidden node of a i-th data point. DISPLAYFORM1 is a convex function in terms of p j, q j, and c. Note that for any p j and q j, there exist θ that forms them, so the following lemma holds. DISPLAYFORM2 Now we introduce the following theorem describing the important properties of the function L g A (θ).Theorem 2.5 The function L g A (θ) has following properties: 1) it is non-convex and non-concave except for the case that activation values are all zeros, 2) every local minimum is a global minimum, 3) every critical point that is not a global minimum is a saddle point. A function f (x, y) = (xy − 1) 2 is not convex, since it has a saddle point at x = y = 0. Similarly, the L g A (θ) is a quadratic function of v j b j, so it is non-convex and nonconcave. If activation values are all zeros, then DISPLAYFORM0, so the global minima are critical points. In other critical points, at least one of the gradients along p j or q j is not zero. If a critical point satisfies ∇ pj L = 0 (or ∇ qj L = 0), then it is a saddle point with respect to w T j and v j (or b j and v j). The detailed proof is in the appendix. To distinguish between the global minimum of L g A (θ) and L f (θ), we introduce subglobal minimum: DISPLAYFORM1 Some of the subglobal minima may not exist in the real loss surface L f (θ). For this kind of regions, there only exist saddle points and the parameter would move to another region by gradient descent methods without getting stuck into local minima. Since the parameter space is a disjoint union of the activation regions, the real loss surface L f (θ) is a piecewise combination of L g A (θ). Using Lemma 2.2 and Theorem 2.5, we conclude as follows: DISPLAYFORM2 The function L f (θ) has following properties: 1) it is non-convex and non-concave, 2) every differentiable local minimum is a subglobal minimum, 3) every critical point that is not a subglobal minimum is a saddle point. We explicitly distinguish differentiable and non-differentiable local minima. The non-differentiable local minima can exist as shown in FIG2. In this section, we answer why poor local minima do exist even in large networks. There are parameter points where all the activation values are zeros eliminating gradient flow . This is a well-known region that forms poor and flat local minima. We introduce another kind of poor region called linear region and show that it always forms poor local minima when a dataset is nonlinear. In a more general setting, we prove that a network has every local minimum of the narrower networks of the same number of layers. There always exists a linear region where all activation values are one, and its subglobal minima stay in that region. This subglobal minimum in an error which is same as that of linear regression, so if given dataset is nonlinear the error would be poor. We can easily spot a linear region by manipulating biases to satisfy w T j x i + b j > 0. One way of achieving this is by selecting b j as: DISPLAYFORM0 To say that the model can get stuck in the linear region, it is necessary to find the subglobal minima in that region. If f (x i, θ) is linear, then it is of form u T, y 2 = 2, x 3 = T, y 3 = 3. The network has one hidden layer and no biases. We increased the number of hidden nodes from one to four. DISPLAYFORM1 The ratio of the poor regions decreases as the size of the network grows. We show it numerically by identifying all subglobal minima of a simple network. For the MNIST , we estimated subglobal minima of randomly selected activation values and compared with the rich regions. Training a neural network is known to be NP-Complete BID1, due to the nonconvexity and infinite parameter space of DNNs. The number of possible combination of activation values has the complexity of O(2 N h) for f (x i, θ), so we restricted the experiments to a small size of hidden layers and datasets to find all subglobal minima in a reasonable time. Consider the Equation 4 again. The subglobal minimum is a solution of the convex optimization for L g A (θ). To compute optimal parameters, we need to solve linear equations DISPLAYFORM0, and ∇ c L g A = 0. For simplicity, we assume that biases are removed, then the gradient ∇ pj L g A is as follows: DISPLAYFORM1 DISPLAYFORM2 As a , the linear equation to solve is as follows: The leftmost matrix in the Equation 7 is a square matrix. If it is not full rank, we compute a particular solution. FIG4 shows four histograms of the poor subglobal minima for the different number of hidden nodes. As shown in the histograms, gradient descent based methods are more likely to avoid poor subglobal minima in larger networks. It also shows that the subglobal minima arise from the smaller networks. Intuitively speaking, adding a node provides a downhill path to the previous poor subglobal minima without hurting the rich subglobal minima in most cases. DISPLAYFORM3 For more realistic networks and datasets, we conducted experiments on MNIST. We used networks of two hidden layers consisting of 2k and k nodes respectively. The networks use biases, softmax outputs, cross entropy loss, mini-batch size of 100, and Adam Optimizer BID9. Assuming that the Corollary 2.7 holds for multilayer networks, the subglobal minima can be estimated by gradient descent methods. It is impossible to compute all of them, so we randomly selected various combinations of activation values with P (a = 1) = P (a = 0) = 0.5. Then we removed rectified linear units and multiplied the fixed activation values as follows: DISPLAYFORM0 where h A is the output of the second hidden layer. The rich subglobal minima were estimated by optimizing the real networks since it would end up in one of the subglobal minima that exist in the real loss surface. The experiments were repeated for 100 times, and then we computed mean and standard deviation. The are shown in TAB0 and it implies that most of the regions in the large networks are rich, whereas the small networks have few rich regions. In other words, it is more likely to end up in a rich subglobal minimum in larger networks.5 RELATED WORKS BID0 proved that linear networks with one hidden layer have the properties of the Theorem 2.5 under minimal assumptions. Recently, BID8 proved that it also holds for deep linear networks. Assuming that the activation values are drawn from independent Bernoulli distribution, a DNN can be mapped to a spin-glass Ising model in which the number of local minima far from the global minima diminishes exponentially with the size of the network BID2. Under same assumptions in BID2, the effect of nonlinear activation values disappears by taking expectation, so nonlinear networks satisfy the same properties of linear networks BID8.Nonlinear DNNs usually do not encounter any significant obstacles on a single smooth slope path BID6 BID4 explained that the training error at local minima seems to be similar to the error at the global minimum which can be understood via random matrix theory. The volume of differentiable sub-optimal local minima is exponentially vanishing in comparison with the same volume of global minima under infinite data points . Although a number of specific example of local minima can be found in DNNs , it seems plausible to state that most of the local minima are near optimal. As the network width increases, we are more likely to meet a random starting point from which there is a continuous, strictly monotonically decreasing path to a global minimum BID14. Similarly, the starting point of the DNNs approximate a rich family of hypotheses precisely BID3. Another explanation is that the level sets of the loss become connected as the network is increasingly overparameterized BID5. These works are analogous to our showing that the parameters would end up in one of the subglobal minima which are similar to the global minima. We conjecture that the loss surface is a disjoint union of activation regions where every local minimum is a subglobal minimum. Using the concept of loss surface decomposition, we studied the existence of poor local minima and experimentally investigated losses of subglobal minima. However, the structure of non-differentiable local minima is not yet well understood yet. These non-differentiable points exist within the boundaries of the activation regions which can be obstacles when using gradient descent methods. Further work is needed to extend knowledge about the local minima, activation regions, their boundaries. Let θ ∈ R A be a differentiable point, so it is not in the boundaries of the activation regions. This implies that w T j x i + b j = 0 for all parameters. Without loss of generality, we assume w T j x i + b j < 0. Then there exist > 0 such that w T j x i + b j + < 0. This implies that small changes in the parameters for any direction does not change the activation region. Since L f (θ) and L g A (θ) are equivalent in the region R A, the local curvatures of these two function around the θ are also the same. Thus, the θ is a local minimum (saddle point) in L f (θ) if and only if it is a local minimum (saddle point) in L g A (θ). DISPLAYFORM0 is a linear transformation of p j, q j, and c, the DISPLAYFORM1 2 is convex in terms of p j, q j, and c. Summation of convex functions is convex, so the lemma holds. A.3 PROOF OF THEOREM 2.5 Assume that activation values are not all zeros, and then consider the following Hessian matrix evaluated from v j and b j for some non-zero activation values a ij > 0: DISPLAYFORM2 Let v j = 0 and b j = 0, then two eigenvalues of the Hessian matrix are as follows: DISPLAYFORM3 There exist c > 0 such that g A (x i, θ) > y i for all i. If we choose such c, then DISPLAYFORM4 ∂vj ∂bj > 0 which implies that two eigenvalues are positive and negative. Since the Hessian matrix is not positive semidefinite nor negative semidefinite, the function L g A (θ) is non-convex and non-concave. We organize some of the gradients as follows: DISPLAYFORM5 We select a critical point θ * where ∇ wj L g A (θ *) = 0, ∇ vj L g A (θ *) = 0, ∇ bj L g A (θ *) = 0, and ∇ c L g A (θ *) = 0 for all j. Case 1) Assume that ∇ pj L g A (θ *) = 0 and ∇ qj L g A (θ *) = 0 for all j. These points are global minima, since ∇ c L g A (θ *) = 0 and L g A (θ) is convex in terms of p j, q j, and c. Case 2) Assume that there exist j such that ∇ pj L g A (θ DISPLAYFORM6 There exist an element w * in w j such that ∇ vj ∇ w * L g A (θ *) = 0. Consider a Hessian matrix evaluated from w * and v j. Analogous to the proof of, this matrix is not positive semidefinite nor negative semidefinite. Thus θ * is a saddle point. Case 3) Assume that there exist j such that ∇ qj L g A (θ *) = 0. Since ∇ bj L g A (θ *) = v j ∇ qj L g A (θ *) = 0, the v j is zero. Analogous to the Case 2, a Hessian matrix evaluated from b j and v j is not positive semidefinite nor negative semidefinite. Thus θ * is a saddle point. As a , every critical point is a global minimum or a saddle point. Since L g A (θ) is a differentiable function, every local minimum is a critical point. Thus every local minimum is a global minimum.

represent this text to find a much much longer article about it with say hundreds-thousands of words Examples:
Provided: "semi-supervised and transfer learning on packet flow classification, via a system of cooperative or adversarial neural blocks" Match: "The fault diagnosis in a modern communication system is traditionally supposed to be difficult, or even impractical for a purely data-driven machine learning approach, for it is a humanmade system of intensive knowledge. A few labeled raw packet streams extracted from fault archive can hardly be sufficient to deduce the intricate logic of underlying protocols. In this paper, we supplement these limited samples with two inexhaustible data sources: the unlabeled records probed from a system in service, and the labeled data simulated in an emulation environment. To transfer their inherent knowledge to the target domain, we construct a directed information flow graph, whose nodes are neural network components consisting of two generators, three discriminators and one classifier, and whose every forward path represents a pair of adversarial optimization goals, in accord with the semi-supervised and transfer learning demands. The multi-headed network can be trained in an alternative approach, at each iteration of which we select one target to update the weights along the path upstream, and refresh the residual layer-wisely to all outputs downstream. The actual show that it can achieve comparable accuracy on classifying Transmission Control Protocol (TCP) streams without deliberate expert features. The solution has relieved operation engineers from massive works of understanding and maintaining rules, and provided a quick solution independent of specific protocols. A telecommunications network is a collection of distributed devices, entirely designed and manufactured by humans for a variety of transmission, control and management tasks, striving to provide a transparent channel between external terminals, via an actual internal relay process node by node. As a typical conversation in the style of client and server, the two linked nodes send their messages in the form of packets, encapsulated the load with miscellaneous attributes in headers to ensure the correctness, consistency, and smoothness of the entire process. A typical header includes packet sequence number, source and destination addresses, control bits, error detection codes, etc. The large-scale network cannot always work ideally, due to its inherent complexity inside massive devices and their interactions. When there is a malfunction of a device, either caused by the traffic overload, or software bugs, or hardware misconfiguration, or malicious attacks, it will be reflected on the packet streams that pass through, such as packet loss, timeout, out of order, etc. System administrators captured those suspicious streams and sent back to the service center for cautious offline analysis, which is time-consuming and domain-specific. The primary challenge of automatic diagnosis is that, it is almost impossible to formalize all the logic inside the system and make them available to artificial intelligence. A typical modern communication system consists of tens of thousands devices end-to-end and runs based on a list of hundreds of protocols layer-by-layer BID6 ). If we could figure out the latent states of protocols by constructing specific features from raw bytes, the subsequent classification tasks would be quite straightforward and easy to implement. For instance, the Transmission Control Protocol (TCP) relies on sequence numbers to judge the receiving order of packets, which may be just big integers roughly linearly growing from the view of machine learning models. Another example is a few critical control bits may reside among much more useless bits, such as checksum codes, which is harmful noises for models. Even we have the patience to dive into all the industrial protocols and build up an exhausted feature library; eventually, we will fail again to achieve the target of automation, one of the main advantages of the modern data-driven approach. Another difficulty is scarce of labeled samples. In spite of there are seemingly numerous packet flows running through the Internet all the time, the real valid faults occur at random and occupy only a tiny portion of whole traffic volume. The actual labeled data are usually collected from the archive of fault cases, which is hard to have enough samples for all possible categories, or cannot at least cover them completely. The previous works on this issue mainly follow two technical routes: 1) a traditional two-phase framework, using expert features and some general-propose classifiers BID1 ); 2) an end-to-end approach based on deep learning for automatic feature extraction . All these prior arts seldom use the generative models, which is usually more promising for expressing structural relationship among random variables. And they may fuse 1-2 data sources in semi-supervised setting , but not scale to even more data sources. In this paper, we resort to a generative model to mimic the messages in a terminals conversation and enrich the target data domain from two abundant but different information sources: labeled but from simulation, and genuine but unlabeled. The transfer and semi-supervised demands are integrated into an intuitive framework, composed of a connected graph of multiple simple Generative Adversarial Networks (GANs)' components, trained in an alternative optimization approach. The contribution of this paper includes: 1) combine three kinds of data sources in a generative approach, to solve the small-sample problem with a simulation environment; 2) extend the two players in usual GANs to a system of multiple ones, still keeping its merit of end-to-end training; 3) verify its effect on our practice problem of packet sequence classification. The left of paper is organized as below: first, we introduce the previous work selectively in network anomaly detection and the research frontier in the generative neural network. Next, we present the model and algorithm in detail with feature design at different levels. The of experiments are followed in Section 4. Finally, we conclude the whole article. The anomaly detection in communication packets has been long-term studied, either for the Quality of Service (QoS) management or instruction detection. The article BID1 summarized the past works based on their applied technologies, which almost cover all popular machine learning methods before 2012.1 The works after that mainly switched to deep learning as it goes popular, which are surveyed by ourselves. In general, all of these keep developing with both two aspects: the level of automation and the ability of models. BID4 started to train neural networks (NN) on labeled data to build models for more categories of anomalies than hard-coded rules, with an expert feature library and three-layer perceptron classifiers. Later, BID0 verified the feasibility of Self Organizing Map (SOM) in an unsupervised scenario, where the unexpected packets were automatically saved for analyzers. BID2 used the online learning to quickly adapt to the newly occurred attacking samples feedbacked by users, without the effort of retraining. BID11 used the self-taught learning to enrich the dataset from unlabeled live stream, and build up a comprehensive feature space for embedding. The enhancement can be observed even using a simple K-Nearest-Neighbors. On the other hand, the neural network models also advance consistently. The early attempts include PCA NNLiu et al. FORMULA3, Wavelet NNSun et al., etc. Yin et al. (2017 used Recurrent Neural Network (RNN) on classifying 5 categories of packet flows, and achieved obviously better than models ignoring the temporal orders. BID16 designed an NN with 6-dimensional manual feature vectors and 3 hidden layers for inherent mapping as input and claimed accuracy improvements after testing. also used self-taught learning, similar to BID11, but with more sophisticated models. It extracted features automatically from unlabeled data by a sparse auto-encoder, and classify them by a single hidden layer perceptron. To our best knowledge, it is the first time we employ the generative neural networks for the semisupervised and transfers learning simultaneously on this problem. The classical GANs compose of two neural components contesting with each other in a zero-sum game BID7 ). It can be extended to more components for the following purposes (but not limited to): 1) reflecting the relationship between multiple random variables. BID12 solved the multi-class classification by adding an extra classifier to denote the conditional probability p(y|X). The newly classifier can shift the burden of predicting labels from identifying fake samples, which is blended in the previous work BID14. This triple-player formulation makes the model even clearer and brings more stableness during training. In multi-view learning, BID3 defined one discriminator for each view and enabled the distribution estimation over all possible output y if any subset of view on a particular input X is fixed. 2) Enhancing the training process. BID8 addressed the mode collapse problem in GANs by training many generators, which envisions to discriminate itself to an additional classifier and fool the original discriminator meanwhile. It improved the steadiness of training significantly. Instead, BID5 used multiple discriminators to ensemble a more powerful one; on the other hand, the training process is retarded by a predefined function (similar to soft-max) on the top of them to match the generator's capability better. In this paper, we only focus on the former purpose. The packet stream in reality is a sequence of attributed events e = {(t i, {a i,j})|i ∈ N, j ∈ N }, where t i is the timestamp sealed by the receiver, and a i,j is a tuple of key-value parsed from packet headers. The label c of e can be K classes of anomalies containing 1 special class for normality. To focus the main aspect of our problem, we prefer to simplify it by two assumptions: 1) anomalies can only happen at one side of the communication, such as server side, to prevent the number of possible situations from blowing up to K 2. It seldom happens that, both terminals have problems simultaneously and produce a complicated interaction that can fully not be diagnosed by a onesided model. In fact, we can train two individual models for both sides, and thus the records from client side can be removed from the train set in experiments.2) The continuous valued (or finegrained in 10 −6 s) timestamps are ignored, while only their ascending index is kept, from 1 to T. We insert dummy packets, which replicate sequence id from the previous one and fill all other fields with 0, to denote an occurence of timeout between two consecutive items. The overall number of dummy packets is informative to models since it indicates how many periods the opposite side has not responded. It is justified because most protocols are indifferent to the exact time intervals during sending/receiving unless they exceed the predefined timeout threshold. The available content of attributes depends on how much effort we want to pay for an accurate inspection of packet headers. The clearer we want to know about the states of every system running, the much we should know about the details of a given protocol. There are 3 levels of feature engineering: 1) raw bytes of headers, which needs little effort; 2) the numerical values (sequence index, integer or Boolean) parsed by a data scheme indicating their positions and necessity; 3) the latent states defined in protocols, based on complete domain knowledge and Finite-State Machine (FSM) -driven analysis. For instance, a packet at level 1 may be only a binary sequence, like 1001... 1000, which is unreadable for humans. At level 2 FIG0, it turns to be a data structure with proper data types, but without semantics defined by the protocol draft. The array of structures can further be arranged into a multi-dimensional vector. At level 3 FIG0, the inherent states are explicitly parsed out, and the sequence finally achieves its most compact representation -a categorical vector. NN can digest all levels above, though an extra effort is needed for discrete values at level 3, discussed later in Sec. 3.2.2. A simplified version of FSM for a sender during TCP's transmission phase. In the typical situation, the sender shuttles between the Ready and Listen state. When there is some error occurred in the direction that data goes, the system will retreat to a previous position and resend the datum; when the acknowledge packet loses in the coming direction, a timeout will be triggered, and the system will resend the last piece of data. In practice, these states are implicitly encoded in the headers, and the computer program has to analyze them based on TCP protocol. Assume that the N -dimensional sequence Y = {y t : t ∈ T} is generated from repetitive applying of a function G on a latent M -dimensional vector z: DISPLAYFORM0 Where c is a categorical variable to switch between the types of anomalies, and θ is the parameters of G. We assume that, if z ∼ N (0, σ 2) and c ∼ Cat(π), the ed Y will conform to a distribution where our observationsŶ come from. In the adversary approach, we guide the training of G by minimizing the distance (cross entropy, etc.) between the p(Y, c) and observedp(Y, c), via the transformation of an extra binary discriminator D r: DISPLAYFORM1 where H is the cross entropy between two distributions. Similarly, we define a function G s that transforms Y to its correspondence Y s in the simulation world, and a function D s to discriminate them from real simulation data: DISPLAYFORM2. For the unlabeled data, we define a D c to distinguish them from the generated samples without labels c: DISPLAYFORM3 Figure 2: The layout of building blocks in our solution. The arrows denote for the information flow among blocks, and they are connected as three forward paths in (a), representing real, simulated and unlabeled information sources colored as orange, green, and blue. The noises driven the whole process is assumed to be sampled from two independent sources, z ∼ N (0, σ 2) and c ∼ Cat(π). The π are estimated directly based on real labeled data. In (b), the white block C is trained with a fixed G, colored with shading.. The overall optimization goal is a convex combination of Eq. 3, 5 and 7: DISPLAYFORM4 where λ s, λ c is the coefficients to adjust the importance of targets. Once we obtain the p(Y, c) in the implicit form of G, the classifier p(c|Y) can be derived with the marginal distribution p(c) according to the Bayes rule. The neural components and their connections are shown in Fig. 2. There are 3 information sinks in Fig. 2a), each of which connects to exact 2 data sources and corresponds to one part of loss function in Eq. 8; minimizing them concurrently, is equivalent to make the data come from different paths look the same with the best effort. Note that the graph G is connected, directed and acyclic, which makes any topological order (must exist based on graph theory) of nodes is viable to a gradient descent approach. It provides a convenience for the design of optimization heuristics in Sec. 3.2.3. The trained G will be frozen in a secondary training process in Fig. 2b ), to consistently offer the classifier C its pseudo-samples with labels until convergence. The neural blocks in Fig. 2 can be built from even more fine-grained modules: 1 an input layer concatenating a vector of z (or y) and an optional one-hot vector for c; 2 a Long Short-Term Memory (LSTM) layer mapping a sequence to a vector; 3 a LSTM layer reversely mapping a vector to a sequence; 4 a fully connected layer; 5 a sigmoid (or softmax) layer classifying real/fake samples (or outputting labels). The generators G is built by 1 + 3, and G s is 2 + 3, and C is 2 + 5; all D, D s, D u share a same structure, which is 2 + 1 + 5. The modules 2, 3 and 4 are kept as one single hidden layer here, after the multi-layer structures have been tried in practice and their improvement was found to be negligible. For the discrete state sequence at feature level 3, it is feasible to map its discrete values into continuous vector globally by Word2vec during preprocessing, since the C is what we finally interested for diagnosis and the intermediate are not visible to end users indeed. We need a heuristics to guide the whole optimization process of G, depicted in Algo. 1. During every mini-batch iterations of training, every time we select a forward path whose loss function contributes most in the Eq. 8, weighted by λs, and update the θ of blocks on the way in the form of gradient descent. The all three sub-loss functions will be updated after G has been modified. Once the process has found a selected target that does not contribute to the overall goal, the update will be rolled back, and the algorithm will switch to others. The failure record for any target will not bring into next batch. For individual blocks, RMSprop is preferred for components containing a recurrent layer, including G, G s and C, while all discriminators use Adam instead. The real labeled data are collected from the archive of fault cases, probed at the core section of a wireless telecom site in service. The problematic TCP streams have 4 categories: 1) uplink packet loss at random, 2) downlink packet loss at random, 3) packet loss periodically when load exceeds capacity, 4) checksum error at random, 5) healthy but false alarmed. The faults in the connection establishment and release phase are not considered here. In the historical records, all we have are 18 samples, unevenly distributed from category 1-5, which are 5, 7, 3, 1, and 2.The unlabeled data are captured from the same site, having 2000 real samples sufficient for training propose. Though its size is much larger than the labeled ones, it can hardly contain valid anomalies by an arbitrary, short-term collection. It provides a reference for the average condition of service quality for a specific site. The simulation is conducted from a mirror environment in labs, with a similar or maybe simplified configuration. The 5 types of anomalies are generated in equal ratio, and each of them has 400 records. The phenomenon of occurring errors here is much more evident than reality: 1) for uplink and 2) downlink packets, the portability of loss is 50%, 3) the stream have 50% of time transmitting over the limit of throughput; 4) the probability of checksum error is 50%.The sequence lengths of all synthetic data are all fixed to 500.The data preprocessing on the TCP header is based on different levels discussed in Sec. 3.1, where the latent states of TCP include normal, resend, and timeout, distilled from 7 useful attributes, and also from 24 bytes of raw binary records. A simple FSM is defined to compress the attributes to several states, according to TCP's standard logic. All sequences are split into the size of 500, and the ones shorter than that are padded by 0. The performance of our multi-class problem is evaluated by the accuracy Acc = N correct /N, which is only metrics cared by operation engineers. They rank all the possible causes descendingly based on the model's output and take the whole list as a recommendation, which acts as a shortcut leading them to the underlying causes much more quickly, especially compared to their existing manualbased approach. We measure two variations of accuracy: averaging on samples as above, and averaging on class, i.e., K c=1 N Nc Acc c, to emphasize the performance on minor-classes. 3-fold cross-validation is used for the set of real labeled data, with all other 2 datasets always assisting the train partition for every fold. The program is based on Keras 2 and a GAN library 3. The one-to-many recurrent layer of Keras is implemented merely in the form of many-to-many, by repeating the input as a vector of equal length as output. The dimension of noise X is set to 20, and hidden dimension of LSTM is 10 with L2 regularization. The learning rates of all components are configured to 10 −3. All other parameters are kept by default. We have two kinds of factors to combine into a workable model: feature levels and data sources, shown in Tab. 1. The solution in Fig. 2a) can be trimmed based on available data sources. In the group 1 of Tab.1, we give two referential models for comparison, one (Line 1) is the dummy model which always give the most frequent class as prediction, and the other (Line 2) is the if we only use the simulation data both for train and test with one standalone classifier. With the deliberately amplified anomalies and evenly distributed classes, the performance can be quite ideal, and to some extent be an empirical upper limit of diagnosis. It can be observed in group 4∼5 that, the simulation data are crucial for substantial enhancement by adding more typical anomalies, while the unlabeled contributes slightly via only supplying normal samples. The improvements in weighted accuracy are more obvious, which is more than twice that of dummy model. On the other hand, the features still act an essential role in our problem. The level 3 feature can always perform better than the rest two levels, while the level 1 can especially not produce any meaningful . The level 2 can approximate the best performance with the help of massive data, offering an always worse but still acceptable accuracy, without the effort of understanding protocols. The evolution of losses of 3 discriminators is demonstrated in FIG2, and the classifier is shown in FIG2 ). All loss curves of GANs' components converge to their roughly horizontal limits, with more or less mutually influenced fluctuations, caused by the continuous attempts to seek for a better equilibrium in every batch. However, these efforts seem to merely make the overall loss tremble, instead of moving to lower places. The variances of simulated data are obviously much smaller than real data, which may be ascribed to the sufficient size of data and evenly distribution among classes, whereas the real data are of imbalance, and have few samples to validate the improvements during training. We terminated the whole process at iteration 10 4, and use the trained G to gain a corresponding C, which is much easier to train as the loss goes steadily to its convergence level, shown in FIG2 ). In this paper, the widely used semi-supervised and transfer learning requirements have been implemented in an integrated way, via a system of cooperative or adversarial neural blocks. Its effectiveness has been verified in our application of packet flow classification, and it is hopeful to be a widely adopted method in this specific domain. The work also prompts us that, complex machine learning tasks and their compound loss functions can be directly mapped into connected networks, and their optimization process can be designed over an entire graph, rather than each individual's hierarchical layers. In future work, we may study how to apply this approach to even larger scale tasks, and make a theoretical analysis of the existence of equilibrium and why we can always reach it."

The loss surface of neural networks is a disjoint union of regions where every local minimum is a global minimum of the corresponding region.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question: Represent the article for finding a single-sentence TLDR

Question: Contextualized word representations, such as ELMo and BERT, were shown to perform well on a various of semantic and structural (syntactic) task. In this work, we tackle the task of unsupervised disentanglement between semantics and structure in neural language representations: we aim to learn a transformation of the contextualized vectors, that discards the lexical semantics, but keeps the structural information. To this end, we automatically generate groups of sentences which are structurally similar but semantically different, and use metric-learning approach to learn a transformation that emphasizes the structural component that is encoded in the vectors. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics. Finally, we demonstrate the utility of our distilled representations by showing that they outperform the original contextualized representations in few-shot parsing setting. Human language 1 is a complex system, involving an intricate interplay between meaning (semantics) and structural rules between words and phrases (syntax). Self-supervised neural sequence models for text trained with a language modeling objective, such as ELMo , BERT , and RoBERTA (b), were shown to produce representations that excel in recovering both structure-related information (; van Schijndel & Linzen; ;) as well as in semantic information . In this work, we study the problem of disentangling structure from semantics in neural language representations: we aim to extract representations that capture the structural function of words and sentences, but which are not sensitive to their content. For example, consider the sentences: We aim to learn a function from contextualized word representations to a space that exposes these similarities. Crucially, we aim to do this in an unsupervised manner: we do not want to inform the process of the kind of structural information we want to obtain. We do this by learning a transformation that attempts to remove the lexical-semantic information in a sentence, while trying to preserve structural properties. Disentangling syntax from lexical semantics in word representations is a desired property for several reasons. From a purely scientific perspective, once disentanglement is achieved, one can better control for confounding factors and analyze the knowledge the model acquires, e.g. attributing the predictions of the model to one factor of variation while controlling for the other. In addition to explaining model predictions, such disentanglement can be useful for the comparison of the representations the model acquires to linguistic knowledge. From a more practical perspective, disentanglement can be a first step toward controlled generation/paraphrasing that considers only aspects of the structure, akin to the style-transfer works in computer vision, i.e., rewriting a sentence while preserving its structural properties while ignoring its meaning, or vice-versa. It can also inform search-based application in which one can search for "similar" texts while controlling various aspects of the desired similarity. To achieve this goal, we begin with the intuition that the structural component in the representation (capturing the form) should remain the same regardless of the lexical semantics of the sentence (the meaning). Rather than beginning with a parsed corpus, we automatically generate a large number of structurally-similar sentences, without presupposing their formal structure (§3.1). This allows us to pose the disentanglement problem as a metric-learning problem: we aim to learn a transformation of the contextualized representation, which is invariant to changes in the lexical semantics within each group of structurally-similar sentences (§3.3). We demonstrate the structural properties captured by the ing representations in several experiments (§4), among them automatic identification of structurally-similar words and few-shot parsing. The problem of disentangling different sources of variation has long been studied in computer vision, and was recently applied to neural models (; ;). Such disentanglement can assist in learning representations that are invariant to specific factors, such as pose-invariant face-recognition or style-invariant digit recognition . From a generative point of view, disentanglement can be used to modify one aspect of the input (e.g., "style"), while keeping the other factors (e.g., "content") intact, as done in neural image style-transfer . In the field of NLP, disentanglement is much less researched. In controlled natural language generation and style transfer, several works attempted to disentangle factors of variation such as sentiment or age of the writer, with the intention to control for those factors and generate new sentences with specific properties, or transfer existing sentences to similar sentences that differ only in the those properties. Several works have trained conditional generative models by explicitly conditioning a decoder network with a vector of attributes. On training, the attributes derive from the training sentence, while in testing the conditioning vector can be set to generate a text with the desired attributes. Other works; aim to achieve style transfer (as opposed to generation) by explicitly training representations that are invariant to the controlled attributes (e.g. by co-training of a generator and attribute discriminator). A decoder generates the transferred text from the disentangled representation and from an explicit attribute representation. use a conditioned back-translation approach to achieve a similar goal. While these works try to disentangle sentence-level attributes, in this study we focus on disentangling between two components in the representations of individual words. Several works examine the way semantic and syntactic information is distributed across the layers of neural models of text . They use diagnostic classifiers to predict syntactic properties and demonstrate that different parts of the model encode information in different levels of abstraction (e.g. POS information, dependency label and semantic role). Liu et al. (2019a) used diagnostic classifiers trained to predict various syntactic and semantic properties from state-of-the-art LMs representations, and demonstrated that many syntactic and semantic distinctions are encoded in the probed representations. probed the attention patterns of BERT, and showed that individual attention-heads focus in syntactically-meaningful relations in the input. Beyond the descriptive level, recent works have focused on supervised extraction of syntax-related representations from neural representations. used a linear transformation, inspired from the notion of "similarity order" in classic word representation learning, to tailor uncontextualized word representations to syntactic vs. semantic tasks. demonstrated that it is possible to train a linear transformation, under which squared euclidean distance between transformed contextualized word vectors correspond to the distances between the respective words in the syntax tree that represents the hierarchical structure of the sentence. Concurrent to this work, have used a variational estimation method of the information-bottleneck principle to extract word embeddings that are useful to the end task of parsing. While impressive, those works presuppose a specific formal syntactic structure (e.g. annotated parse trees following a specific linguistic annotation schema) and use this syntactic signal to learn structural information in a supervised manner. In other words, these works assume a given structure, and use supervision to make the structural information more salient, mapping (or forcing) the neural representations to known linguistic properties. In contrast, we aim to expose the structural information encoded in the network in an unsupervised manner, without pre-supposing an existing syntactic annotation scheme. Our goal is to learn a function f: R n → R m, which operates on contextualized word representations x and extracts vectors f (x) which make the structural information encoded in x more salient, while discarding as much lexical information as possible. In the sentences "Maple syrup is delicious" and "Neural networks are interesting", we want to learn a f such that f (v Moreover, we would like the relation between the words "maple" and "delicious" in the second sentence, to be similar to the relation between "neural" and "interesting" in the first sentence: pair(v . Operativly, we represent pairs of words (x, y) by the difference between their transformation f (x) − f (y), and aim to learn f that preserves:. The choice to represent pairs this way was inspired by several works that demonstrated that nontrivial semantic and syntactic relations between uncontextualized word representations can be approximated by simple vector arithmetic (a; b;). To learn f, we start with groups of sentences that the sentences within each group are known to share their structure but differ in their lexical semantics. We call the sentences in each group structurally equivalent. Figure 1 shows an example of two structurally equivalent sets. Acquiring such sets is challenging, especially if we do not assume a known syntactic formalism and cannot mine for sentences based on their observed tree structures. To this end, we automatically generate the sets starting with known sentences and sampling variants from a language model (§3.1). Our sentenceset generation procedure ensures that words from the same set that share an index also share their structural function. We call such words corresponding. Figure 1: Two groups of structurally-equivalent sentences. In each group, the first sentence is original sentence from Wikipedia, and the sentences below it were generated by the process of repeated BERT substitution. Some sets of corresponding words-that is, words that share the same structural function-are highlighted in the same color. We now proceed to learn a function f to map contextualized vectors of corresponding words (and the relations between them, as described above) to neighbouring points in the space. We train f such that the representation assigned to positive pairs -pairs that share indices and come from the same equivalent set -is distinguished from the representations of negative pairs -challenging pairs that come from different sentences, and thus do not share the structure of the original pair, but can, potentially, share their lexical meaning. We do so using Triplet loss, which pushes the representations of pairs coming from the same group closer together (§3.3). Figure 2 sketches the network. Figure 2: An illustration of triplet-loss calculation. Pairs of words are represented by the difference between their transformation f, which is identical for all words. The pairs of words in the anchor and positive sentences are lexically different, but structurally similar. The negative example presented here is especially challenging, as it is lexically similar, but structurally different. In order to generate sentences that approximately share their structure, we sequentially replace content words in the sentence with other content words, while aiming to maintain the grammatically of the sentence, and keep its structure intact. Replacing words with words of the same POS, as done in , does not answer to those requirements, as this method does not respect various restrictions that apply to words that share different function within the sentence. For example, verb argument structure dictates limitations on the arguments the predicate receives, and verbs differ in properties such as whether or not they accept a complement. Replacing verbs with other verbs does not guarantee fulfilling these limitations. Since we do not want to rely on syntactic annotation (apart from the level of POS tags) when performing this replacement, we opted to use a pre-trained language model -BERT -under the assumption that strong neural language models do implicitly encode many of the syntactic restrictions that apply to words in different grammatical functions (e.g., we assume that BERT would not predict a transitive verb in the place of an intransitive verb, or a verb that accepts a complement in the place of a verb that does not accept a complement). While this assumption seems to hold with regard to basic distinctions such as transitive vs. intransitive verbs, its validity is less clear in the more nuanced cases, in which small differences in the surface level can translate to substantial differences in the deep structuresuch as replacing a control verb with a raising verb. This is a limitation of the current approach, although we find that the average sentence we generate is grammatical and similar in structure to the original sentence. Moreover, as our goal is to expose the structural similarity encoded in neural language models, we find it reasonable to only capture the distinctions that are captured by a state-of-the-art neural language model. Implementation Concretely, we rely on a BERT masked LM model. We start each group with a Wikipedia sentence, for which we generate k = 6 equivalent sentences by iterating over the sentence from left to right sequentially, masking the ith word, and replacing it with one of BERT's top-30 predictions. Crucially, to increase semantic variability, we perform the replacement in place (online), that is, after randomly choosing a guess w, we insert w to the sentence at index i, and continue guessing the i + 1 word based on the modified sentence. 5 We exclude a closed set of a few dozens of words (mostly function words) and keep them unchanged in all k variations of a sentence. To the extent that BERT learns to recover corrupted sentences by suggesting replacements that respect the probability distribution of actual natural language, the suggestions would be both semantically and structurally correct. We further maintain structural correctness by maintaining the POS, and encourage semantic diversity by the auto-regressive replacement process. We find that the average sentence is grammatical and maintain the structure of the original sentence, although some generated sentences violate these requirements. In the appendix §?? we show some additional generated groups, and highlight some recurring errors. The sets in Figure 1 were generated using this method. We use the method to generate N = 150, 000 equivalent sets E i of structurally equivalent sentences, and collect the contextualized vector representations of words in these sets, ing in 1,500,000 training pairs and 200,000 evaluation pairs for the training process of f. We experiment with both ELMo and BERT-based contextualized representations. In average, we sample 11 pairs from each group of equivalent sentences. For ELMo, we represent each word in context as a concatenation of the last two ELMo layers (excluding the word embedding layer, which is not contextualized and therefore irrelevant for structure), ing in representations of dimension 2048. For BERT, we concatenate the mean of the words' representation across all 23 layers of BERT-Large, with the representation of layer 16, which was found by most indicative of syntax. We learn the mapping function f using triplet loss (Figure 2). Concretely, given a group of equivalent sentences E i, we randomly choose two sentences to be the anchor sentence S A, and the positive sentence S P, and sample two different word indices {i 1, i 2}. We represent pairs as their differences after transformation, ing in the anchor pair V A and positive pair V P: where f is the parameterized syntactic transformation we aim to learn. We also consider a negative pair: coming from sentence S N which is not in the equivalent set. As f has shared parameters for both words in the pair, it can thus be considered a part of a Siamese network, making our learning procedure an instance of a triplet Siamese network. We choose f to be a simple model: a single linear layer that maps from dimensionality 2048 to 75. The dimensional of the transformation was chosen according to development set performance. We use triplet loss to move the representation of the anchor vector V A closer to the representation of the positive vector V P and farther apart from the representation of the negative vector V N. , we calculate the softmax version of the triplet loss: where dist(x, y) = 1 − x y x y is the cosine-distance between the vectors x and y. Note that → 0, as expected. The triplet objective is optimized end-to-end using the Adam optimizer . We train for 5 epochs with a mini-batch of size 500 6, and take the last model as the final syntactic extractor. During training, the gradient backpropagates through the pair vectors to the parameters f of the Siamese model, to get representations of individual words that are similar for corresponding words in equivalent sentences. We note that we do not back-propagate the gradient to the contextualized vectors: we keep them intact, and only adjust the learned transformation. Hard negative sampling We obtain the negative vectors V N using hard negative sampling. For each mini-batch B, we collect 500 {V A i, V P i} pairs, each pair taken from an equivalent set E i. The negative instances V N i are obtained by searching the batch for a vector that is closest to the anchor and comes from a different set: where dist is again the cosine distance. In addition, we enforce a symmetry between the anchor and positive vectors, by adding a pair (positive, anchor) for each pair (anchor, positive) in B. That is, V N i is the "most misleading" word-pair vector: it comes from a sentence that has a different structure than the structure of V We have trained the syntactic transformation f in a way that should encourage it to retain the structural information encoded in contextualized vectors, but discard other information. We assess the representations our model acquired in an unsupervised manner, by evaluating the extant to which the local neighbors of each transformed contextualized vector f (x) share known structural properties, such as grammatical function within the sentence. For the baseline, we expect the neighbors of each vector to share a mix of semantic and syntactic properties. For the transformed vectors, we expect the neighbors to share mainly syntactic properties. Finally, we demonstrate that in a fewshot setting, our representations outperform the original ELMO representation, indicating they are indeed distilled from syntax, and discard other information that is encoded in ELMO vectors but is irrelevant for the extraction of the structure of a sentence. Corpus For training the transformation f, we rely on 150,000 sentences from Wikipedia, tokenized and POS-tagged by spaCy 8. The POS tags are used in the equivalent set generation to filter replacement words. Apart from POS tagging, we do not rely on any syntactic annotation during training. The evaluation sentences for the experiments mentioned below are sampled from a collection of 1,000,000 original and unmodified Wikipedia sentences (different from those used in the model training). Figure 3 shows a 2-dimensional t-SNE projection of 15,000 random content words. The left panel projects the original ELMo states, while the right panel is the syntactically transformed ones. The points are colored according to the dependency label (relation to parent) of the corresponding word, assigned by the spacy parser. As can be seen, in the original ELMo representation most states -apart from those characterized by a specific part-of-speech, such as amod (adjectives, in orange) or nummod (numbers, in light green) -do not fit well into a single cluster. In contrast, the syntactically transformed vectors are more neatly clustered, with some clusters, such as direct objects (brown) and prepositional-objects (blue), that are relatively separated after, but not before, the transformation. Interestingly, some functions that used to be a single group in ELMo (like the adjectives in orange, or the noun-compounds in 6 A large enough mini-batch is necessary to find challenging negative examples. 7 We implicitly assume that any pair coming from a different group of equivalent sentences is a valid negative example -that is, does not share the structural relation that exists between the anchor pair's words. This is a relatively mild assumption, as due to sparsity, in high probability two different sentences do not share the very same structure 8 https://spacy.io/ ELMo Transformed Figure 3 : t-SNE projection of ELMO states, colored by syntactic function, before (left) and after (right) the syntactic transformation. green) are now split into several clusters, corresponding to their use in different sentence positions, separating for examples adjectives that are used in subject positions from those in object position or within prepositional phrases. Additionally, as noun compounds ("maple" in "maple syrup") and adjectival modifiers ("tasty" in "tasty syrup") are relatively structurally similar (they appear between determiners and nouns within noun phrases, and can move with the noun phrase to different positions), they are split and grouped together in the representation (the green and orange clouds). To quantify the difference, we run K-means clustering on the projected vectors, and calculate the average cluster purity score as the relative proportion of the most common dependency label in each cluster. The higher this value is, the more the division to clusters reflect division to grammatical functions (dependency labels). We run the clustering with different K values: 10, 20, 40, 80. We find an increase in class purity following our transformation: from scores of 22.6%, 26.8%, 32.6% and 36.4% (respectively) for the original vectors, to scores of 24.3%, 33.4%, 42.1% and 48.0% (respectively) for the transformed vectors. Examples Below are a few query words (Q) and their closest neighbours before (N) and after (NT) the transformation. Note the high structural similarity of the entire sentence, as well as the function of the word within it (Q1: last word of subject NP in a middle clause, Q2: possessed noun in sentence initial subject NP, Q3: head of relative clause of a direct object): Q:in this way of thinking, an impacting projectile goes into an ice-rich layer -but no further. N:they generally have a pre-engraved rifling band to engage the rifled launch tube, spin-stabilizing the projectile, hence the term " rifle ". NT:to achieve a large explosive yield, a linear implosion weapon needs more material, about 13 kgs. Q: the mint's director at the time, nicolas peinado, was also an architect and made the initial plans. N: the director is angry at crazy loop and glares at him, even trying to get a woman to kick crazy loop out of the show (which goes unsuccessfully). NT: jetley's mother, kaushaliya rani, was the daughter of high court advocate shivram jhingan. Q: their first project is software that lets players connect the company's controller to their device N: you could try use norton safe web, which lets you enter a website and show whether there seems to be anything bad in it. NT: the city offers a route-finding website that allows users to map personalized bike routes We expect our transformed vectors to capture more structural and less lexical similarities than the source vectors. We expect each vectors' neighbors in space to share the structural function of the word over which the vector was collected, but not necessarily share its lexical meaning. We focus on the following structural properties: Table 1: Results in the closest-word queries, before and after the application of the syntactic transformation. "Basline" refers to unmodified ELMo vectors, "Transformed" refers to ELMo vectors after the learned syntactic transformation f, and "Transformed-untrained" refers to ElMo vectors, after a transformation that was trained on a randomely-initialized ELMo. "Difficult" refers to evaluation on the subset of POS tags which are most structurally diverse. • Dependency-tree edge of a given word (dep-edge), that represents its function (subject, object etc.) • The dependency edge of the word parent's (head's dep-edge) in the tree -to represent higher level structure, such as a subject that resides within a relative clause, as in the word'man" in the phrase "the child that the man saw". • Depth in the dependency tree (distance from the root of the sentence tree). • Constituency-parse paths: Consider, for example, the sentence "They saw the moon with the telescope". The word "Telescope" is a part of a noun-phrase "The telescope", which resides inside a prepositional phrase "with the telescope", which is part of the Verbal phrase ""Saw with the telescope". The complete constituency path for this word is therefore "NP-PP-VP". We calculate the complete tree path to the root (Tree-path-complete), as well as paths limited to lengths 2 and 3. For this evaluation, we parse 400,000 random sentences taken from the 1-million-sentences Wikipedia sample, run ELMo and BERT to collect the contextualized representations of the sentences' words, and randomly choose 400,000 query word vectors (excluding function words). We then retrieve, for each query vector x, the value vector y that is closest to x in cosine-distance, and record the percentage of closest-vector pairs (x, y) that share each of the structural properties listed above. For the tree depth property, we calculate the Pearson correlation between the depths of the queries and the retrieved values. We use Spacy parser for dependency-parsing, and the Berkeley Neural Parser for constituency parsing. We exclude function words from the evaluation. Easier and Harder cases The baseline models tend to retrieve words that are lexically similar. Since certain words tend to appear at above-chance probability in certain structural functions, this can make the baseline be "right for the wrong reason", as the success in the closest-word test reflects lexical similarity, rather than grammatical generalization of the model. To control for this confounding, we sort the different POS tags according to the entropy of their dependency-labels distribution, and repeat the evaluation only for words belonging to those POS tags having the highest entropy (those POS tags are the most structurally variant, and tend to appear in different structural functions). We find that the performance of the baselines (ELMo, BERT models) on those words drops significantly, while the performance of our model are only mildly influenced, further indicating the superiority of our model in capturing structural rather than lexical information. The for ELMo are presented in Table 4. For BERT, we witnessed similar, but somewhat lower, accuracy: for example, 68.1% dependency-edge accuracy, 56.5% head's dependencyedge accuracy, and 22.1% complete constituency-path accuracy. The for BERT are available in the appendix §C, and for the reminder of the paper, we focus in ELMo. We observe significant improvement over the baseline for all tests. The correlation between the depth in tree of the query and the value words, for examples, rises from 44.8% to 56.1%, indicating that our model encourages the structural property of the depth of the word to be more saliently encoded in its representation compared with the baseline. The most notable relative improvement is recorded with regard to full constituency-path to the root: from 16.6% before the structural transformation, to 25.3% after itan improvement of 52%. In addition to the increase in syntax-related properties, we observe a sharp drop -from 73.6% to 28.4% --in the proportion of query-value pairs that are lexically identical (lexical match, Table 4). This indicates our transformation f removes much of the lexical information, which is irrelevant for structure. To assess to what extent the improvements stems from the information encoded in ELMo, rather than being an artifact of the triplet-loss training, we also evaluate on a transformation f that was trained on a randomly-initialized ELMo, a surprisingly strong baseline . We find this model performs substantially worse than the baseline (Table 4, "Transformed-untrained (all)"). The absolute nearest-neighbour accuracy values may appear to be relatively low: for example, only 67.6% of the (query, value) pairs share the same dependency edge. As the model acquires its representation without being exposed to human-mandated syntactic convention, some of the apparent discrepancies in nearest neighbours may be due to the fact the model acquires different kind of generalization, or learned a representation that emphasizes different kinds of similarities. Still, we expect the ing (75 dimensional) representations to contain distilled structure information that is mappable to human notions of syntax. To test this, we compare dependency-parsers trained on our representation and on the source representation. If our representation indeed captures structural information, we expect it to excel on a low data regime. To this end, we test our hypothesis with few-shot dependency parsing setup, where we train a model to predict syntactic trees representation with only a few hundred labeled examples. We use an off-the-shelf dependency parser and first swap the pre-trained Glove embeddings with ELMo contextualized embeddings . In order to have a fair comparison with our method, we use the concatenation of the two last layers of Elmo; we refer to this experiment as elmo. As our representation is much smaller than ELMo's (75 as opposed to 2048), a potential issue for a low data regime is the high parameter number to optimize in the later case, therefore a lower dimension can achieve better . We design two additional baseline experiments to remedy this potential issue: Using PCA in order to reduce the representation dimensionality. We randomly chose 1M words from Wikipedia, calculated their representation with ELMo embeddings and performed PCA. This transformation is applied during training on top of ELMo representation while keeping the 75 first components. This experiment is referred to as elmo-pca. This representation should perform well if the most salient information in the ELMo representations are structural. We exepct it to not be the case. Automatically learning a matrix that reduces the embedding dimension. This matrix is learned during training and can potentially extract the relevant structural information from the representations. We refer to this experiment as elmo-reduced. Lastly, we examine the performance of our representation, where we apply our structural extraction method on top of ELMo representation. We refer to this experiment as syntax. We run the few-shot setup with multiple training size values: 50, 100, 200, 500. The -for both labeled (LAS) and unlabeled (UAS) attachment scores-are presented in Figure 4, and the numerical are available in the appendix §B. We notice that in the lower training size regime, we obtain the best performances compared to all baselines. The more training data is used, the gap between our representation and the baselines reduced, but the syntax representation still outperforms elmo. Reducing the dimensions with PCA (elmo-pca) works considerably worse than ELMo, indicating that the most salient information is indeed not structural, and the PCA loses important information. Reducing the dimensions with a learned matrix (elmo-reduced) works substantially better than ELMo, and achieve the same UAS as our representation from 200 training sentences onward. However, our transformation was learned in an unsupervised fashion, without access to the syntactic trees. Finally, when considering the labeled attachment score, where the model is tasked at predicting not only the child-parent relation but also its label, our syntax representation outperforms elmo-reduced. In this work, we propose an unsupervised method for the distillation of structural information from neural contextualized word representations. We used a process of sequential BERT-based substitu- Figure 4: Results of the few shot parsing setup tion to create a large number of sentences which are structurally similar, but semantically different. By controlling for one aspect -structure -while changing the other -lexical choice, we learn a metric (via triplet loss) under which pairs of words that come from structurally-similar sentences are close in space. We demonstrated that the representations acquired by this method share structural properties with their neighbors in space, and show that with a minimal supervision, those representations outperform ELMo in the task of few-shots parsing. The method presented here is a first step towards a better disentanglement between various kinds of information that is represented in neural sequence models. The method used to create the structurally equivalent sentences can be useful by its own for other goals, such as augmenting parse-tree banks (which are often scarce and require large resources to annotate). In a future work, we aim to extend this method to allow for a more soft alignment between structurally-equivalent sentences. Table 4: Results in the closest-word queries, before and after the application of the syntactic transformation. "Basline" refers to unmodified vectors derived from BERT, and "Transformed" refers to the vectors after the learned syntactic transformation f. "Difficult" refers to evaluation on the subset of POS tags which are most structurally diverse.

Represent the TLDR for finding the corresponding scientific article

We distill language models representations for syntax by unsupervised metric learning

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.