Question: Represent the article for finding a single-sentence TLDR

Question: Estimating the importance of each atom in a molecule is one of the most appealing and challenging problems in chemistry, physics, and material engineering. The most common way to estimate the atomic importance is to compute the electronic structure using density-functional theory (DFT), and then to interpret it using domain knowledge of human experts. However, this conventional approach is impractical to the large molecular database because DFT calculation requires huge computation, specifically, O(n^4) time complexity w.r.t. the number of electrons in a molecule. Furthermore, the calculation should be interpreted by the human experts to estimate the atomic importance in terms of the target molecular property. To tackle this problem, we first exploit machine learning-based approach for the atomic importance estimation. To this end, we propose reverse self-attention on graph neural networks and integrate it with graph-based molecular description. Our method provides an efficiently-automated and target-directed way to estimate the atomic importance without any domain knowledge on chemistry and physics. In molecules, each atom has the importance in manifesting the entire molecular properties, and estimating such atomic importance plays a key role in interpreting molecular systems. For these reasons, the atomic importance estimation has been consistently studied in the scientific communities (; ;). However, estimating the atomic importance is one of the most challenging tasks in chemistry and quantum mechanics because the importance of each atom is comprehensively determined based on atomic properties, neighbor atoms, bonding types, and target molecular property. The most common approach for estimating the atomic importance is to interpret the electronic structure using density-function theory (DFT) . In this approach, the atomic importance is estimated through three steps: 1) A human expert selects appropriate functional and basis sets for a given molecule to apply DFT; 2) The electronic structure of the molecule is calculated based on DFT calculation; 3) The human expert estimates the atomic importance by interpreting the calculated electronic structure in terms of target molecular property. Although some methods are developed to estimate relative contributions of atoms in molecules, their generality is typically limited to the molecular properties . For this reason, DFT that can generate a general description of the molecule has been most widely used to interpret the molecular systems and to reveal important atoms for target molecular property (; b;). However, the conventional approach based on DFT has three fundamental limitations in efficiency, automation, and generality. • Efficiency: As an example of the electronic structure computations, DFT calculation requires O(n 4) time complexity to compute the electronic structure, where n is the number of basis functions that describe electrons in the molecule. Generally, molecules have more electrons than atoms. • Automation: DFT cannot automatically generate all target-specified physical properties in principle, so human expert should manually select additional calculation method to com-pute target molecular property from the electronic distributions. That is, domain knowledge of the human experts is necessarily required to estimate the atomic importance in terms of the target molecular property. • Generality: For some molecular properties, the relationship between them and the electronic distributions is not clear. Moreover, sometimes the estimation is impossible because the relationships between molecular property and molecular structure are not interpretable. For these limitations, estimating the atomic importance is remaining as one of the most challenging problems in both science and engineering such as physics, chemistry, pharmacy, and material engineering. To overcome the limitations of the conventional approach in estimating the atomic importance, we first exploit machine learning-based approach. To this end, we propose a new concept of reverse self-attention and integrate it with the graph neural networks. The self-attention mechanism was originally designed to determine important elements within the input data to accurately predict its corresponding target or label in natural language processing . Similarly, in graph neural networks, self-attention is used to determine important neighbor nodes within the input graph to generate more accurate node or graph embeddings . Our reverse self-attention is defined as the inverse of the self-attention to calculate how important a selected node is considered in the graph. For a given molecule and target property, the proposed estimation method selects the atom that has the largest reverse self-attention score as the most important atom. The proposed method estimates the target-directed atomic importance through two steps: 1) For the given molecular graphs and their corresponding target properties, self-attention-based graph neural network is trained; 2) After the training, the reverse self-attention scores are calculated, and then the atomic importance is estimated according to the reverse self-attention scores. As shown in this estimation process, neither huge computation nor human experts in chemistry and physics is required. Thus, the proposed method provides an efficient and fully-automated way to estimate the atomic importance in terms of the target molecular property via target-aware training of the graph self-attention. To validate the effectiveness of the proposed method, we conducted comprehensive experiments and evaluated the estimation performance using both quantitative and qualitative analyses. The contributions of this paper are summarized as: • This paper first proposes a machine learning-based approach to estimate the atomic importance in the molecule. • The proposed method drastically reduced the computational cost for the atomic importance estimation from O(n 4) time complexity to the practical time complexity of the graph-based deep learning. • The proposed method provides a fully-automated and target-directed way to estimate the atomic importance. • We comprehensively validated the effectiveness of the proposed method using both quantitative and qualitative evaluations. However, since none of a labeled dataset for the atomic importance estimation and a systematic way to quantitatively evaluate the estimation accuracy, we devised a systematic quantitative evaluation method and validated the effectiveness of the proposed method using it. Before describing the atomic importance estimation based on the reverse self-attention, we briefly two essential concepts for understanding the proposed method in this section: 1) graph-based molecular analysis; 2) graph self-attention and graph attention network. Graph-based molecular machine learning has attracted significant attention from both scientific and machine learning communities. It has shown state-of-the-art performance in various scientific applications such as molecular or crystal property prediction (; ; Lu Figure 1: Overall process of the graph-based molecular machine learning. In the molecular graph, three and two-dimensional vectors mean atom-features and bond-features, respectively. et al., 2019), molecular generation ), and atomic reaction analysis . In the graph-based molecular machine learning, a molecule is represented as an undirected feature graph G = (V, A, X, U), where V is a set of atoms (nodes); A ∈ {0, 1} |V|×|V| is an adjacency matrix indicating the existence of the atomic bonds (edges); X ∈ R |V|×d is a ddimensional atom-feature matrix; U ∈ R |B|×m is a m-dimensional bond-feature matrix; and B is a set of atomic bonds. For a given labeled molecular dataset D = {(G 1, y 1),..., (G |D|, y |D|)}, the goal of the graph-based molecular machine learning is to build a model f: G → y, where y can be target value, class label, or another molecular graph. In this paper, we will refer atom and bond as node and edge, respectively. Fig. 1 shows the overall process of the graph-based molecular machine learning. First, molecular or crystal structures formatted by.xyz or.cif are converted into the molecular graph with the adjacency matrix A, node-feature matrix X, and edge-feature matrix U. Then, graph neural network generates the graph-level embedding of the input molecular graph through the aggregation layers and predict the corresponding target y. In the aggregation layer of the graph neural network, each node in the graph is converted into the node-embedding vector, and it is stacked to the node-embedding matrix in column-wise. The output node-embedding matrix of the k th aggregation layer, H (k), is calculated by: where ψ is an aggregation function, and W (k) is a trainable weight matrix of the k th aggregation layer. Note that H is the input node-feature matrix X. For example, in graph convolutional network (GCN) , H (k), is calculated by: whereÃ is an adjacency matrix containing self-loop and can be calculated byÃ = A + I. After the node-embedding, the node-embedding matrix of the last aggregation layer is converted into the graph-level embedding vector by readout function . For the implementation of the readout function, mean or max-based operations are commonly used (a; . Finally, the target corresponding the input graph is predicted by interpreting the output graph-level embedding vector through the fully-connected layers. The attention and the self-attention mechanisms are originally introduced in natural language processing to refer other data and elements to improve prediction and classification accuracies, respectively . In graph neural networks based on neighbor node aggregation approach such as GCN, the self-attention mechanism is used to calculate the importance of neighbor nodes in generating the node-embedding vectors. Graph attention network (GAT) is the first neural network to apply the self-attention mechanism in graph-based deep learning. By exploiting the self-attention mechanism, GAT showed highly improved prediction and classification accuracies (; . In the k th aggregation layer of GAT, the attention coefficient between node i and its neighbor node j is defined by: where f is a feedforward neural network; V is a trainable weight matrix; and ⊕ is a vector concatenation. Based on the attention coefficient, the attention score, α (k) ij, is calculated by: where N(i) is a set of neighbor nodes of node i in the graph. Finally, in the k th aggregation layer of GAT, the node-embedding vector of node i is calculated by: In GAT, the aggregation layer that generates the node-embeddings based on the graph self-attention is called graph attention layer. In this section, we explain our machine learning-based approach for estimating the atomic importance. To devise the machine learning-based importance estimator, we define a new concept of reverse self-attention and integrate it with GAT. The self-attention mechanism in the molecular graph provides numerical importance of each neighbor atom in terms of target molecular property. To exploit the concept of the self-attention for estimating the atomic importance, reverse self-attention in the k th graph attention layer is defined as the inverse of the self-attention: ρ That is, the reverse self-attention of node i means how much attention node i receives from its neighbor nodes. Fig. 2 shows an example of the reverse self-attention in the graph with five nodes. In this example, the reverse self-attention of n 1 is calculated as the sum of α 21 + α 31 + α 41. As shown in the definition of the reverse self-attention, the additional time and space complexities of the reverse self-attention are negligible because calculating the reverse self-attention is just the sum of pre-computed self-attention scores in prediction or classification time. In GAT, the self-attention scores of the k th graph attention layer are calculated based the nodeembeddings of the (k − 1) th graph attention layer. That is, the self-attention scores of the first layer are calculated by considering 1-hop neighbor nodes, and the second layer's self-attention scores are determined by considering 2-hop neighbor nodes. Thus, the reverse self-attention in the first Figure 3: Overall process of building machine learning-based atomic importance estimator with the reverse self-attention mechanism and selecting important atom or group of atoms using the estimator. graph attention layer indicates the importance of each atom itself, and the second layer's reverse self-attention means the importance of each atom group consists of 1-hop neighbor atoms, not an atom itself. These interpretations can be extended to the atomic importance for the group of atoms that consists of k-hop neighbor atoms by the reverse self-attention in the (k + 1) th graph attention layer. This section explains the way to build a fully-automated and target-directed atomic importance estimator based on the reverse self-attention and GAT. We call this estimator Machine Intelligencebased Atomic Importance Estimator (MIAIE). Fig. 3 shows overall process of building MIAIE and estimating the target-directed atomic importance via MIAIE. The atomic importance estimation using MIAIE consists of four steps: • Step 1: Train GAT for regression or classification on the molecular dataset with the target molecular property. In this training process, GAT automatically learns the self-attentions scores in terms of predicting the target molecular property. • Step 2: Predict the target property of the interesting molecule and extract self-attention scores for each graph attention layer. • Step 3: Calculate the reverse self-attention scores using Eq. and sort them for each graph attention layer. • Step 4: Select atom or group of atoms that have the largest reverse self-attention score as the most important element in the molecule in terms of the target property. As shown in the overall process of estimating the atomic importance using MIAIE, the estimation process is fully-automated and does not require any domain knowledge of the human experts in chemistry and physics. MIAIE is also incomparably efficient than the conventional approach using DFT calculations with O(n 4) time complexity because it uses only the graph neural networks to describe the molecules, and the time complexity of the estimation process (step 3 and 4) is negligible. To accurately validate the effectiveness of MIAIE, we conducted both quantitative and qualitative evaluations on two well-known molecular datasets. However, to the best of our knowledge, neither a labeled dataset for the atomic importance estimation nor a systematic way to evaluate the performance of the atomic importance estimator exists. For this reason, we devised a validation method to quantitatively evaluate the performance of the atomic importance estimators. We will explain this validation method in Section 4.3. We used MolView 1 to visualize the estimation of MIAIE. For the experiments, we used two well-known molecular datasets: Quantum Mechanics9 (QM9) and Estimated SOLubility (ESOL) . These datasets were randomly split into 90% train and 10% test subsets because the test dataset for them are not provided. In the experiments, we validated the effectiveness of MIAIE by splitting train and test datasets to evaluate the generalization capability as well as the atomic importance estimation accuracy. However, we can use MIAIE by fitting it to a given dataset only without considering the generalization capability-for example, training without L 2 -regularization , dropout , or batch normalization if our analysis is only focused on specific molecular datasets containing the interesting molecules. To quantitatively evaluate the effectiveness of MIAIE, QM9 dataset is used. It is a subset of GDB-17 database , in which the structural information of the molecules is given in Cartesian coordinates. QM9 dataset contains 133,886 organic molecules and several target molecular properties such as highest occupied molecular orbital (HOMO), lowest unoccupied molecular orbital (LUMO), and their gap (HOMO-LUMO gap). In the experiments, we used HOMO-LUMO gap as the target molecular property because it is an essential property describing the molecular systems and one of the difficult properties to be interpreted from the molecular structure. Molecules in QM9 dataset have 0.0245∼0.6221 HOMO-LUMO gaps. For the qualitative analysis, we used ESOL dataset that contains aqueous solubility as a target property because the important atoms in terms of aqueous solubility can be easily determined by a human expert. ESOL dataset was originally published with the aqueous solubility of 2,874 compounds but a smaller subset of 1,128 compounds is recently used in chemistry , and we also used the subset of ESOL for the experiment. Unlike QM9 dataset, the structural information of the molecules in ESOL is provided by SMILES representation that does not present hydrogen. For this reason, hydrogen in the molecule is ignored in estimating the atomic importance. We implemented MIAIE using PyTorch 2 and PyTorch Deep Graph Library (DGL) 3. GAT in MI-AIE is also implemented based on the neural network modules of DGL. In the experiments, we used GAT with two graph attention layers and two fully-connected layers. To generate graphlevel embedding, mean-based readout function is applied. Mean squared error (MAE) was used as an objective function to train GAT, and L 2 -regularization with 0.001 regularization coefficient was applied to improve the generalization capability of the model. As a training algorithm, Adam SGD was used with an initial learning rate of 0.001 to fit model parameters of GAT. To accelerate the training and improve the prediction performance of GAT, we concatenated additional molecular features about molecular weight and the number of rings to the graphlevel embedding vector. The source code of MIAIE and the experiment scripts are available at http://----------------------(open after the review). To quantitatively evaluate the effectiveness of MIAIE, we assume that if the selected atom or group of atoms are truly important in terms of the target property, then the gap between the target properties of the original molecule and its selected sub-molecule (atom or group of atoms) will be small. Based: An original molecule made of the carbon ring and its selected sub-molecule that consists of 1-hop neighbor atoms. Note that hydrogens are automatically attached to make a valid molecular structure. The group of atoms that is marked by a red mask means the most important atomic group estimated by MIAIE. White node: hydrogen (H); Gray node: carbon (C). on this assumption, we define atomic importance estimation error (AIEE) as: where G i is a sub-molecule of G i selected by an atomic importance estimator; DF T means the value of the target property of G i computed by DFT calculations; and size(G i) is the number of atoms in G i. Since DFT can accurately calculate the value of the molecular properties in most cases even though it requires huge computation, we used DFT to calculate the value of the target molecular property of unknown G i. In AIEE, the error of each molecule is divided by the number of atoms in the molecule. It is an essential part of AIEE to accurately measure the target property gaps because the larger the size of the original molecule, the greater the probability of increasing the gap between the properties of the original molecule and its smaller sub-molecule with a fixed size. In this experiment, we quantitatively evaluated the effectiveness of MIAIE using AIEE in Eq. on QM9 dataset. We estimated the atomic importance using MIAIE with the reverse self-attention of the second graph attention layer for 13,3889 test molecules in terms of HOMO-LUMO gap and observed 0.00544 on AIEE. This error is relatively small when considering the fact that the molecular properties of the original molecule cannot be preserved completely in sub-molecules. However, we observed a relatively large error of 0.01007 compared to the mean of error (0.00544) on the molecule made of the carbon ring as shown in Fig. 4. This large error is caused due to the chemical characteristics of HOMO-LUMO gap that it is determined by the overall electronic distributions in ring-shaped molecules with the same atoms such as Fig. 4-(a). Thus, this large error in HOMO-LUMO gap is inevitable in splitting a ring-shaped molecule with the same atoms into the sub-molecules because each atom has similar importance. It is a limitation of our validation method for the quantitative evaluation. One possible future work is to modify the estimation step of MIAIE to answer "each atom has similar importance" instead of selecting important atom for the molecules like Fig. 4 -(a). For ESOL dataset, we conducted a qualitative evaluation because the important atom or group of atoms in terms of aqueous solubility can be easily determined by human experts. Furthermore, we denoted the normalized atomic importance of the most important group of atoms, which is selected by MIAIE. Due to the limitation of the space, we present only three natural and two interesting estimation in this paper. can be easily justified: 1) Since nitrogen and oxygen exist only in the selected sub-molecule, the estimation of 2-Nitropropane is natural; 2) Similar to the of 2-Nitropropane, MIAIE accurately selected a group of atoms that contains the largest number of nitrogen and oxygen in Metronidazole; 3) In Indoline, only three sub-molecules that consists of 1-hop neighbor atoms can contain the nitrogen. However, among these sub-molecules, the selected group of atoms has the largest electronic charge that improve the reactivity to water, so the estimation of MIAIE is also chemically resonable. Some interesting are and (e). Benfluralin in Fig. 5 -(d) has a low aqueous solubility of -5.53 in log-scale because the C-CF 3 group is attached . Interestingly, although we did not tell this chemical observation about the relationship between the C-CF 3 group and aqueous solubility to MIAIE, it exactly selected the C-CF 3 group as the most important group of atoms in terms of aqueous solubility. Another interesting is Coumaphos in Fig. 5 -(e). In this molecule, the C-CO 2 group was selected as the most important group of atoms rather than PO 3 S group that contains the largest number of oxygen. However, this estimation is chemically reasonable because the carbons surrounding the PO 3 S group prevent the reaction between the PO 3 S group and water. In these estimation , we observed that MIAIE can estimate the atomic importance by considering complex chemical rules related to the functional properties of the atomic groups and the overall environment of the molecules, even though it is trained without any domain knowledge of chemistry and quantum mechanics. To check the correctness of the estimation of MIAIE, we also measured the atomic importance on the molecules that have the symmetric structure because the structurally-equivalent groups of atoms must have the same atomic importance. Fig. 6 shows the estimation of MIAIE for the molecules containing the symmetric structure. Benfluralin has a locally-symmetric structure in two C-NO 2 groups, and MIAIE correctly estimated the atomic importance of two C-NO 2 groups as the same value. Similarly, the C 24 O 4 molecule has globally-symmetric structure throughout the overall molecular structure. For the C 24 O 4 molecule, MIAIE correctly estimated two C-CO 2 groups as the most important group of atoms with the same score. Figure 6: Two original molecules with symmetric structure and their selected sub-molecules in terms of aqueous solubility. The group of atoms that is marked by a blue mask means the second most important atomic group estimated by MIAIE. This paper first exploited machine learning approach to estimate the atomic importance in molecules. To this end, the reverse self-attention was proposed and integrated with graph attention network. The proposed method is efficient and fully-automated. Furthermore, it can estimate the atomic importance in terms of the given target molecular property without human experts. However, the proposed method can estimate the importance of the group of atoms that consists of k-hop neighbor atoms only, even though some important group of atoms may have an arbitrary shape such as ring and bar. As the future work, it is necessary to modify the proposed method to estimate the importance of the group of atoms with arbitrary shape. Fig. 7 shows the original molecules and their selected sub-molecules with the extremely small error. As shown in the , even though the molecules have carbon rings, the important group of atoms was correctly captured because the molecules are characterized by nitrogen or oxygen. On the other hand, Fig. 8 shows the original molecules and their sub-molecules with the extremely large error. Chemically, double bond plays an important role in determining HOMO-LUMO gap. However, as shown in the of Fig. 8, sub-structures that do not contain the double bonds are selected as an important group of atoms. Thus, we need to develop a descriptor for the molecules that can emphasis the bond-features more strongly.

represent the tldr for finding the corresponding scientific article.

We first propose a fully-automated and target-directed atomic importance estimator based on the graph neural networks and a new concept of reverse self-attention.

Represent the TLDR for finding the corresponding scientific article:

A novel approach to graph classification based on spectral graph convolutional networks and its extension to multigraphs with learnable relations and hierarchical structure. We show state-of-the art results on chemical, social and image datasets.


Question: Represent the scientific article to find a short TLDR

Question: Spectral clustering is a leading and popular technique in unsupervised data analysis. Two of its major limitations are scalability and generalization of the spectral embedding (i.e., out-of-sample-extension). In this paper we introduce a deep learning approach to spectral clustering that overcomes the above shortcomings. Our network, which we call SpectralNet, learns a map that embeds input data points into the eigenspace of their associated graph Laplacian matrix and subsequently clusters them. We train SpectralNet using a procedure that involves constrained stochastic optimization. Stochastic optimization allows it to scale to large datasets, while the constraints, which are implemented using a special purpose output layer, allow us to keep the network output orthogonal. Moreover, the map learned by SpectralNet naturally generalizes the spectral embedding to unseen data points. To further improve the quality of the clustering, we replace the standard pairwise Gaussian affinities with affinities leaned from unlabeled data using a Siamese network. Additional improvement can be achieved by applying the network to code representations produced, e.g., by standard autoencoders. Our end-to-end learning procedure is fully unsupervised. In addition, we apply VC dimension theory to derive a lower bound on the size of SpectralNet. State-of-the-art clustering are reported for both the MNIST and Reuters datasets. Discovering clusters in unlabeled data is a task of significant scientific and practical value. With technological progress images, texts, and other types of data are acquired in large numbers. Their labeling, however, is often expensive, tedious, or requires expert knowledge. Clustering techniques provide useful tools to analyze such data and to reveal its underlying structure. Spectral Clustering BID20 BID16 BID22 ) is a leading and highly popular clustering algorithm. It works by embedding the data in the eigenspace of the Laplacian matrix, derived from the pairwise similarities between data points, and applying k-means to this representation to obtain the clusters. Several properties make spectral clustering appealing: First, its embedding optimizes a natural cost function, minimizing pairwise distances between similar data points; moreover, this optimal embedding can be found analytically. Second, spectral clustering variants arise as relaxations of graph balanced-cut problems BID22. Third, spectral clustering was shown to outperform other popular clustering algorithms such as k-means DCN, VaDE, DEPICT and IMSAT (bottom) on simulated datasets in 2D and 3D. Our approach successfully finds these non-convex clusters, whereas the competing algorithms fail on all five examples. (The full set of for these algorithms is shown in FIG4 in Appendix A.) BID22, arguably due to its ability to handle non-convex clusters. Finally, it has a solid probabilistic interpretation, since the Euclidean distance in the embedding space is equal to a diffusion distance, which, informally, measures the time it takes probability mass to transfer between points, via all the other points in the dataset BID15 BID5.While spectral embedding of data points can be achieved by a simple eigen-decomposition of their graph Laplacian matrix, with large datasets direct computation of eigenvectors may be prohibitive. Moreover, generalizing a spectral embedding to unseen data points, a task commonly referred to as out-of-sample-extension (OOSE), is a non-trivial task; see, for example, BID1 BID2 BID9 BID6 ).In this work we introduce SpectralNet, a deep learning approach to spectral clustering, which addresses the scalability and OOSE problems pointed above. Specifically, SpectralNet is trained in a stochastic fashion, which allows it to scale. Moreover, once trained, it provides a function, implemented as a feed-forward network, that maps each input data point to its spectral embedding coordinates. This map can easily be applied to new test data. Unlike optimization of standard deep learning models, SpectralNet is trained using constrained optimization, where the constraint (orthogonality of the net outputs) is enforced by adding a linear layer, whose weights are set by the QR decomposition of its inputs. In addition, as good affinity functions are crucial for the success of spectral clustering, rather than using the common Euclidean distance to compute Gaussian affinity, we show how Siamese networks can be trained from the given unlabeled data to learn more informative pairwise distances and consequently significantly improve the quality of the clustering. Further improvement can be achieved if our network is applied to transformed data obtained by an autoencoder (AE). On the theoretical front, we utilize VC-dimension theory to derive a lower bound on the size of neural networks that compute spectral clustering. Our experiments indicate that our network indeed approximates the Laplacian eigenvectors well, allowing the network to cluster challenging non-convex point sets, which recent deep network based methods fail to handle; see examples in Figure 1. Finally, SpetralNet achieves competitive performance on MNIST handwritten digit dataset and state-of-the-art on the Reuters document dataset, whose size makes standard spectral clustering inapplicable. Recent deep learning approaches to clustering largely attempt to learn a code for the input that is amenable to clustering according to either the k-means or mixture of gaussians clustering models. DCN BID24 directly optimizes a loss composed of a reconstruction term (for the code) and the k-means functional. DEC BID23 iteratively updates a target distribution to sharpen cluster associations. DEPICT adds a regularization term that prefers balanced clusters. All three methods are pre-trained as autoencoders, while DEPICT also initializes its target distribution using k-means or other standard clustering algorithms. Several other recent approaches rely on a variational autoencoder that utilizes a Gaussian mixture prior, see, for example, VaDE BID27 and GMVAE BID7. IMSAT BID11 is based on data augmentation, where the net is trained to maximize the mutual information between inputs and predicted clusters, while regularizing the net so that the cluster assignment of original data points will be consistent with the assignment of augmented points. Different approaches are proposed by BID4, who uses a deep belief net followed by non-parametric maximum margin clustering (NMMC), and by BID25, who introduce a recurrent-agglomerative framework to image clustering. While these approaches achieve accurate clustering on standard datasets (such as the MNIST and Reuters), the use of the k-means criterion, as well as the Gaussian mixture prior, seems to introduce an implicit bias towards the formation of clusters with convex shapes. This limitation seems to hold even in code space. This bias is demonstrated in FIG0 (bottom), which shows the failure of several of the above approaches on relatively simple clustering tasks. In contrast, as is indicated in FIG0 (top), our SpectralNet approach appears to be less vulnerable to such bias. The full set of runs can be found in Appendix A.In the context of spectral clustering, BID21 learn an autoencoder that maps the rows of a graph Laplacian matrix onto the corresponding spectral embedding, and then use k-means in code space to cluster the underlying data. Unlike our work, which learns to map an input data point to its spectral embedding, Tian et al.'s network takes as input an entire row of the graph Laplacian, and therefore OOSE is impractical, as it requires to compute the affinities of each new data point to all the training data. Also of interest is the kernel spectral method by BID0, which allows for out of sample extension and handles large datasets through smart sampling (but does not use a neural network). BID26 address the problem of 3D shape segmentation. Their work, which focuses on learning graph convolutions, uses a graph spectral embedding through eigenvector decomposition, which is not learned. In addition, we enforce orthogonalization stochastically through a constraint layer, while they attempt to learn orthogonalized functional maps by adding an orthogonalization term to the loss function, which involves non-trivial balancing between two loss components. Other deep learning works use a spectral approach in the context of supervised learning. BID12 apply supervised metric learning, showing that their method approximates the eigenvectors of a 0-1 affinity matrix constructed from the true labels. BID13 trained a network to compute graph Laplacian eigenvectors using supervised regression. Their approach, however, requires the true eigenvectors for training, and hence does not easily scale to large datasets. Finally, a number of papers showed that stochastic gradient descent can be used effectively to compute the principal components of covariance matrices, see, e.g., BID19 and references therein. The setup in these papers assumes that in each iteration a noisy estimate of the entire input matrix is provided. In contrast, in our work we use in each iteration only a small submatrix of the affinity matrix, corresponding to a small minibatch. In future work, we plan to examine how these algorithms can be adapted to improve the convergence rate of our proposed network. In this section we present our proposed approach, describe its key components, and explain its connection to spectral clustering. Consider the following standard clustering setup: Let X = {x 1, . . ., x n} ⊆ R d denote a collection of unlabeled data points drawn from some unknown distribution D; given a target number of clusters k and a distance measure between points, the goal is to learn a similarity measure between points in X and use it to learn a map that assigns each of x 1,..., x n to one of k possible clusters, so that similar points tend to be grouped in the same cluster. As in classification tasks we further aim to use the learned map to determine the cluster assignments of new, yet unseen, points drawn from D. Such out-of-sample-extension is based solely on the learned map, and requires neither computation of similarities between the new points and the training points nor re-clustering of combined data. In this work we propose SpectralNet, a neural network approach for spectral clustering. Once trained, SpectralNet computes a map F θ: R d → R k and a cluster assignment function c: R k → {1, . . ., k}. It maps each input point x to an output y = F θ (x) and provides its cluster assignment c(y). The spectral map F θ is implemented using a neural network, and the parameter vector θ denotes the network weights. The training of SpectralNet consists of three components: (i) unsupervised learning of an affinity given the input distance measure, via a Siamese network (see Section 3.2); (ii) unsupervised learning of the map F θ by optimizing a spectral clustering objective while enforcing orthogonality (see Section 3.1); (iii) learning the cluster assignments, by k-means clustering in the embedded space. In this section we describe the main learning step in SpectralNet, component (ii) above. To this end, let w: DISPLAYFORM0 be a symmetric affinity function, such that w(x, x) expresses the similarity between x and x. Given w, we would like points x, x which are similar to each other (i.e., with large w(x, x)) to be embedded close to each other. Hence, we define the loss DISPLAYFORM1 where y, y ∈ R k, the expectation is taken with respect to pairs of i.i.d. elements (x, x) drawn from D, and θ denotes the parameters of the map y = F θ (x). Clearly, the loss L SpectralNet (θ) can be minimized by mapping all points to the same output vector (F θ (x) = y 0 for all x). To prevent this, we require that the outputs will be orthonormal in expectation with respect to D, i.e., DISPLAYFORM2 As the distribution D is unknown, we replace the expectations in and by their empirical analogues. Furthermore, we perform the optimization in a stochastic fashion. Specifically, at each iteration we randomly sample a minibatch of m samples, which without loss of generality we denote x 1,..., x m ∈ X, and organize them in an m × d matrix X whose ith row contains x T i. We then minimize the loss DISPLAYFORM3 where y i = F θ (x i) and W is a m × m matrix such that W i,j = w(x i, x j). The analogue of for a small minibatch is DISPLAYFORM4 where Y is a m × k matrix of the outputs whose ith row is y T i. We implement the map F θ as a general neural network whose last layer enforces the orthogonality constraint. This layer gets input from k units, and acts as a linear layer with k outputs, where the weights are set to orthogonalize the output Y for the minibatch X. LetỸ denote the m × k matrix containing the inputs to this layer for X (i.e., the outputs of F θ over the minibatch before orthogonalization). A linear map that orthogonalizes the columns ofỸ is computed through its QR decomposition. Specifically, for any matrix A such that A T A is full rank, one can obtain the QR decomposition via the Cholesky decomposition A T A = LL T, where L is a lower triangular matrix, and then setting Q = A L −1 T. This is verified in Appendix B. Therefore, in order to orthogonalizeỸ, the last layer multipliesỸ from the right by DISPLAYFORM5, whereL is obtained from the Cholesky decomposition ofỸ TỸ and the √ m factor is needed to satisfy.We train this spectral map in a coordinate descent fashion, where we alternate between orthogonalization and gradient steps. Each of these steps uses a different minibatch (possibly of different sizes), sampled uniformly from the training set X. In each orthogonalization step we use the QR decomposition to tune the weights of the last layer. In each gradient step we tune the remaining weights using standard backpropagation. Once SpectralNet is trained, all the weights are freezed, including those of the last layer, which simply acts as a linear layer. Finally, to obtain the cluster assignments c 1,... c 2, we propagate x 1,... x n through it to obtain the embeddings y 1,..., y n ∈ R k, and perform k-means on them, obtaining k cluster centers, as in standard spectral clustering. These algorithmic steps are summarized below in Algorithm 1 in Sec. 3.3. The loss can also be written as DISPLAYFORM0 The symmetric, positive semidefinite matrix D − W forms the (unnormalized) graph Laplacian of the minibatch x 1,..., x m. For k = 1 the loss is minimized when y is the eigenvector of D − W corresponding to the smallest eigenvalue. Similarly, for general k, under the constraint, the minimum is attained when the column space of Y is the subspace of the k eigenvectors corresponding to the smallest k eigenvalues of D − W. Note that this subspace includes the constant vector whose inclusion does not affect the final cluster assignment. Hence, SpectralNet approximates spectral clustering, where the main differences are that the training is done in a stochastic fashion, and that the orthogonality constraint with respect to the full dataset X holds only approximately. SpectralNet therefore trades accuracy with scalability and generalization ability. Specifically, while its outputs are an approximation of the true eigenvectors, the stochastic training enables its scalability and thus allows one to cluster large datasets that are prohibitive for standard spectral clustering. Moreover, once trained, SpectralNet provides a parametric function whose image for the training points is (approximately) the eigenvectors of the graph Laplacian. This function can now naturally embed new test points, which were not present at training time. Our experiments with the MNIST dataset (Section 5) indicate that the outputs of SpectralNet closely approximate the true eigenvectors. Finally, as in common spectral clustering applications, cluster assignments are determined by applying k-means to the embeddings y 1,... y n. We note that the k-means step can be replaced by other clustering algorithms. Our preference to use k-means is based on the interpretation (for normalized Laplacian matrices) of the Euclidean distance in the embedding space as diffusion distance in the input space BID15 BID5. In spectral clustering, the symmetric normalized graph Laplacian DISPLAYFORM0 2 can use as an alternative to the unnormalized Laplacian D − W. In order to train SpectralNet with normalized graph Laplacian, the loss function should be replaced by DISPLAYFORM1 DISPLAYFORM2. Batch size considerations Typically in classification or regression, the loss is a sum over the losses of individual examples. In contrast, SpectralNet loss is summed over pairs of points, and each summand describes relationships between data points. This relation is encoded by the full n×n affinity matrix W full (which we never compute explicitly). The minibatch size m should therefore be sufficiently large to capture the structure of the data. For this reason, it is also highly important that minibatches will be sampled at random from the entire dataset at each step, and not be fixed across epochs. When the minibatches are fixed, the knowledge of W full is reduced to a (possibly permuted) diagonal sequence of m × m blocks, thus ignoring many of the entries of W full. In addition, while the output layer orthogonalizesỸ, we do not have any guarantees on how well it orthogonalizes other random minibatches. However, in our experiments we observed that if m is large enough, it approximately orthogonalizes other batches as well, and its weights stabilize as training progresses. Therefore, to train SpectralNet, we use larger minibatches compared to common choices made by practitioners in the context of classification. In our experiments we use minibatches of size 1024 for MNIST and 2048 for Reuters, re-sampled randomly at every step. Choosing a good affinity measure is crucial to the success of spectral clustering. In many applications, practitioners use an affinity measure that is positive for a set of nearest neighbor pairs, combined with a Gaussian kernel with some scale σ > 0, e.g., DISPLAYFORM0 where one typically symmetrizes W, for example, by setting DISPLAYFORM1 Euclidean distance may be overly simplistic measure of similarity; seeking methods that can capture more complex similarity relations might turn out advantageous. Siamese nets BID10 BID17 are trained to learn affinity relations between data points; we empirically found that the unsupervised application of a Siamese net to determine the distances often improves the quality of the clustering. Siamese nets are typically trained on a collection of similar (positive) and dissimilar (negative) pairs of data points. When labeled data are available, such pairs can be chosen based on label information (i.e., pairs of points with the same label are considered positive, while pairs of points with different labels are considered negative). Here we focus on datasets that are unlabeled. In this case we can learn the affinities directly from Euclidean proximity or from graph distance, e.g., by "labeling" points x i, x j positive if x i − x j is small and negative otherwise. In our experiments, we construct positive pairs from the nearest neighbors of each point. Negative pairs are constructed from points with larger distances. This Siamese network, therefore, is trained to learn an adaptive nearest neighbor metric. A Siamese net maps every data point x i into an embedding z i = G θsiamese (x i) in some space. The net is typically trained to minimize contrastive loss, defined as DISPLAYFORM2 where c is a margin (typically set to 1).Once the Siamese net is trained, we use it to define a batch affinity matrix W for the training of SpectralNet, by replacing the Euclidean distance x i − x j in with z i − z j.Remarkably, despite being trained in an unsupervised fashion on a training set constructed from relatively naive nearest neighbor relations, in Section 5 we show that affinities that use the Siamese distances yield dramatically improved clustering quality over affinities that use Euclidean distances. This implies that unsupervised training of Siamese nets can lead to learning useful and rich affinity relations. Our end-to-end training approach is summarized in Algorithm 1.Input: X ⊆ R d, number of clusters k, batch size m; Output: embeddings y 1,..., y n, y i ∈ R k, cluster assignments c 1,... c n, c i ∈ {1, . . . k} Construct a training set of positive and negative pairs for the Siamese network; Train a Siamese network; Randomly initialize the network weights θ; while L SpectralNet (θ) not converged do Orthogonalization step: Sample a random minibatch X of size m; Forward propagate X and compute inputs to orthogonalization layerỸ; Compute the Cholesky factorization LL T =Ỹ TỸ;Set the weights of the orthogonalization layer to be √ m L −1 T; Gradient step: Sample a random minibatch x 1,..., x m; Compute the m × m affinity matrix W using the Siamese net; Forward propagate x 1,..., x m to get y 1,..., y m; Compute the loss or; Use the gradient of L SpectralNet (θ) to tune all F θ weights, except those of the output layer; end Forward propagate x 1,..., x n and obtain F θ outputs y 1,..., y n; Run k-means on y 1,..., y n to determine cluster centers;Algorithm 1: SpectralNet training Once SpectralNet is trained, computing the embeddings of new test points (i.e., out-of-sampleextension) and their cluster assignments is straightforward: we simply propagate each test point x i through the network F θ to obtain their embeddings y i, and assign the point to its nearest centroid, where the centroids were computed using k-means on the training data, at the last line of Algorithm 1. Given a dataset X, one can either apply SpectralNet in the original input space, or in a code space (obtained, for example, by an autoencoder). A code space representation is typically lower dimensional, and often contains less nuisance information (i.e., information on which an appropriate similarity measure should not depend). Following BID24 BID23 BID27 and others, we empirically observed that SpectralNet performs best in code space. Unlike these works, which use an autoencoder as an initialization for their clustering networks, we use the code as our data representation and apply SpectralNet directly in that space, (i.e., we do not change the code space while training SpectralNet). In our experiments, we use code spaces obtained from publicly available autoencoders trained by BID27 on the MNIST and Reuters datasets. Our proposed SpectralNet not only determines cluster assignments in training, as clustering algorithms commonly do, but it also produces a map that can generalize to unseen data points at test time. Given a training set with n points, it is thus natural to ask how large should such a network be to represent this spectral map. The theory of VC-dimension can provide useful worst-case bounds for this size. In this section, we use the VC dimension theory to study the minimal size a neural network should have in order to compute spectral clustering for k = 2. Specifically, we consider the class of functions that map all training points to binary values, determined by thresholding at zero the eigenvector of the graph Laplacian with the second smallest eigenvalue. We denote this class of binary classifiers F spectral clustering n. Note that with k = 2, k-means can be replaced by thresholding of the second smallest eigenvector, albeit not necessarily at zero. We are interested in the minimal number of weights and neurons required to allow the net to compute such functions, assuming the affinities decay exponentially with the Euclidean distance. We do so by studying the VC dimension of function classes obtained by performing spectral clustering on n points in arbitrary Euclidean spaces R d, with d ≥ 3. We will make no assumption on the underlying distribution of the points. In the main of this section, we prove a lower bound on the VC dimension of spectral clustering, which is linear in the number of points n. In contrast, the VC dimension of k-means, for example, depends solely on the dimension d, but not on n, hence making k-means significantly less expressive than spectral clustering 1. As a of our main theorem, we bound from below the number of weights and neurons in any net that is required to compute Laplacian eigenvectors. The reader might find the analysis in this section interesting in its own right. Our main shows that for data in R d with d ≥ 3, the VC dimension of F spectral clustering n is linear in the number n of points, making spectral clustering almost as rich as arbitrary clustering of the n points. The formal proof of Theorem 4.1 is deferred to Appendix C. Below we informally sketch its principles. We show that for any integer n (assuming for simplicity that n is divisible by 10), there exists a set of m = n/10 points in R d that is shattered by F spectral clustering n. In particular, we show this for the set of m points placed in a 2-dimensional grid in R d. We then show that for any arbitrary dichotomy of these m points, we can augment the set of points to a larger set X, containing n = 10m points, with a balanced partition of X into two disjoint sets S and T that respects the dichotomy of the 1 For two clusters in R d, k-means clustering partitions the data using a linear separation. It is well known that the VC dimension of the class of linear classifiers in R d is d + 1. Hence, k-means can shatter at most d + 1 points in R d, regardless of the size n of the dataset.original m points. The larger set has the special properties: within S (and resp. T), there is a path between any two points such that the distances between all pairs of consecutive points along the path are small, and all pairs (s, t) ∈ S × T are far apart. We complete the proof by constructing a Gaussian affinity W with a suitable value of σ and showing that the minimizer of the spectral clustering loss for (X, W) (i.e., the second eigenvector of the Laplacian), when thresholded at 0, separates S from T, and hence respects the original dichotomy. By connecting Theorem 4.1 with known regarding the VC dimension of neural nets, see, e.g., BID18, we can bound the size from below (in terms of number of weights and neurons) of any neural net that computes spectral clustering. This is formalized in the following corollary. Corollary 4.2.1. For the class of neural nets with |v| sigmoid nodes and |w| weights to represent all functions realizable by spectral clustering (i.e., second eigenvector of the Laplacian, thresholded at 0) on n points, it is necessary to have |w| 2 |v| 2 ≥ O(n).2. For the class of neural nets with |w| weights from a finite family (e.g., single-precision weights) to represent all functions realizable by spectral clustering, it is necessary to have |w| ≥ O(n).Proof.1. The VC dimension of the class of neural nets with |v| sigmoid units and |w| weights is at most O(|w| 2 |v| 2) . Hence, if |w| 2 |v| 2 < O(n), such net cannot shatter any collection of points of size O(n). From Theorem 4.1, F spectral clustering n shatters at least O(n) points. Therefore, in order for a class of networks to be able to express any function that can be computed using spectral clustering, it is a necessary (but not sufficient) condition to satisfy |w| 2 |v| 2 ≥ O(n).2. The VC dimension of the class of neural nets with |w| weights from a finite family is O(w) . The arguments above imply that |w| ≥ O(n).Corollary 4.2 implies that in the general case (i.e., without assuming any structure on the n data points), to perform spectral clustering, the size of the net has to grow with n. However, when the data has some geometric structure, the net size can be much smaller. Indeed, in a related , the ability of neural networks to learn eigenvectors of Laplacian matrices was demonstrated both empirically and theoretically by BID13. They proved that there exist networks which approximate the eigenfunctions of manifold Laplacians arbitrarily well (where the size of the network depends on the desired error and the parameters of the manifold, but not on n). To numerically evaluate the accuracy of the clustering, we use two commonly used measures, the unsupervised clustering accuracy (ACC), and the normalized mutual information (NMI). For completeness, we define ACC and NMI below, and refer the reader to BID3 for more details. For data point x i, let l i and c i denote its true label and predicted cluster, respectively. Let l = (l 1, . . . l n) and similarly c = (c 1, . . . c n).ACC is defined as DISPLAYFORM0 where Π is the collection of all permutations of {1, . . . k}. The optimal permutation π can be computed using the Kuhn-Munkres algorithm BID14 BID23. (* *) reported in BID24, (†) reported in BID27, (‡) reported in BID8, († †) reported in BID25, (‡ ‡) reported in BID11. The IMSAT on Reuters was obtained on a subset of 10,000 from the full dataset. NMI is defined as DISPLAYFORM1 where I(l; c) denotes the mutual information between l and c, and H(·) denotes their entropy. Both ACC and NMI are in, with higher values indicating better correspondence the clusters and the true labels. We compare SpectralNet to several deep learning-based clustering approaches on two real world datasets. In all runs we assume the number of clusters is given (k=10 in MNIST and k=4 in Reuters).As a reference, we also report the performance of k-means and (standard) spectral clustering. Specifically, we compare SpectralNet to DEC BID23, DCN BID24, VaDE BID27, JULE BID25, DEPICT , and IMSAT BID11. The for these six methods are reported in the corresponding papers. Technical details regarding the application of k-means and spectral clustering appear in Appendix D.We considered two variants of Gaussian affinity functions: using Euclidean distances, and Siamese distances; the latter case follows Algorithm 1. In all experiments we used the loss. In addition, we report of SpectralNet (and the Siamese net) in both input space and code space. The code spaces are obtained using the publicly available autoencoders which are used to pre-train the weights of VaDE 2, and are 10-dimensional. We refer the reader to Appendix D for technical details about the architectures and training procedures. MNIST is a collection of 70,000 28 × 28 gray-scale images of handwritten digits, divided to training and test sets. To construct positive pairs for the Siamese net, we paired each instance with its two nearest neighbors. An equal number of negative pairs were chosen randomly from non-neighboring points. Table 1 shows the performance of the various clustering algorithms on the MNIST dataset, using all 70,000 images for training. As can be seen, the performance of SpectralNet is significantly improved when using Siamese distance instead of Euclidean distance, and when the data is represented in code space rather than in pixel space. With these two components, SpectralNet outperforms DEC, DCN, VaDE, DEPICT and JULE, and is competitive with IMSAT.To evaluate how well the outputs of SpectralNet approximate the true eigenvectors of the graph Laplacian, we compute the Grassmann distance between the subspace of SpectralNet outputs and that of the true eigenvectors. The squared Grassmann distance measures the sum of squared sines of the angles between two k-dimensional subspaces; the distance is in [0, k]. FIG2 shows the Grassmann distance on the MNIST dataset as a function of the training time (expressed as number of parameter updates). It can be seen that the distance decreases rapidly at the beginning of training and stabilizes around 0.026 as time progresses. To check the generalization ability of SpectralNet to new test points, we repeated the experiment, this time training SpectralNet only on the training set, and predicting the labels of the test examples by passing them through the net and associating each test example with the nearest centroid from the k-means that were performed on the embedding of the training examples. The accuracy on test examples was.970, implying that SpectralNet generalizes well to unseen test data in this case. We similarly also evaluated the generalization performance of k-means. The accuracy of k-means on the test set is.546 when using the input space and.776 when using the code space, both significantly inferior to SpectralNet. The Reuters dataset is a collection of English news, labeled by category. Like DEC and VaDE, we used the following categories: corporate/industrial, government/social, markets, and economics as labels and discarded all documents with multiple labels. Each article is represented by a tfidf vector, using the 2000 most frequent words. The dataset contains n = 685, 071 documents. Performing vanilla spectral clustering on a dataset of this size in a standard way is prohibitive. The AE used to map the data to code space was trained based on a random subset of 10,000 samples from the full dataset. To construct positive pairs for the Siamese net, we randomly sampled 300,000 examples from the entire dataset, and paired each one with a random neighbor from its 3000 nearest neighbors. An equal number of negative pairs was obtained by randomly pairing each point with one of the remaining points. Table 1 shows the performance of the various algorithms on the Reuters dataset. Overall, we see similar behavior to what we observed on MNIST: SpectralNet outperforms all other methods, and performs best in code space, and using Siamese affinity. Our SpectralNet implementation took less than 20 minutes to learn the spectral map on this dataset, using a GeForce GTX 1080 GPU. For comparison, computing the top four eigenvectors of the Laplacian matrix of the complete data, needed for spectral clustering, took over 100 minutes using ARPACK. Note that both SpectralNet and spectral clustering require pre-computed nearest neighbor graph. Moreover, spectral clustering using the ARPACK eigenvectors failed to produce reasonable clustering. This illustrates the robustness of our method in contrast to the well known instability of spectral clustering to outliers. To evaluate the generalization ability of SpectralNet, we divided the data randomly to a 90%-10% split, re-trained the Siamese net and SpectralNet on the larger subset, and predicted the labels of the smaller subset. The test accuracy was 0.798, implying that as on MNIST, SpectralNet generalizes well to new examples. We have introduced SpectralNet, a deep learning approach for approximate spectral clustering. The stochastic training of SpectralNet allows us to scale to larger datasets than what vanilla spectral clustering can handle, and the parametric map obtained from the net enables straightforward out of sample extension. In addition, we propose to use unsupervised Siamese networks to compute distances, and empirically show that this in better performance, comparing to standard Euclidean distances. Further improvement are achieved by applying our network to code representations produced with a standard stacked autoencoder. We present a novel analysis of the VC dimension of spectral clustering, and derive a lower bound on the size of neural nets that compute it. In addition, we report state of the art on two benchmark datasets, and show that SpectralNet outperforms existing methods when the clusters cannot be contained in non overlapping convex shapes. We believe the integration of spectral clustering with deep learning provides a useful tool for unsupervised deep learning. To compare SpectralNet to spectral clustering, we consider a simple dataset of 1500 points in two dimensions, containing two nested'C'-shaped clusters. We applied spectral clustering to the dataset by computing the eigenvectors of the unnormalized graph Laplacian L = D − W corresponding to the two smallest eigenvalues, and then applying k-means (with k=2) to these eigenvector embeddings. The affinity matrix W was computed using DISPLAYFORM0, where the scale σ was set to be the median distance between a point to its 3rd neighbor -a standard practice in diffusion applications. FIG3 shows the clustering of the data obtained by SpectralNet, standard spectral clustering, and k-means. It can be seen that both SpectralNet and spectral clustering identify the correct cluster structure, while k-means fails to do so. Moreover, despite the stochastic training, the net outputs closely approximate the two true eigenvectors of W with smallest eigenvalues. Indeed the Grassmann distance between the net outputs and the true eigenvectors approaches zero as the loss decreases. In the next experiment, we trained, DCN, VaDE, DEPICT (using agglomerative clustering initialization) and IMSAT (using adversarial perturbations for data augmentation) on the 2D datasets of FIG0. The experiments were performed using the code published by the authors of each paper. For each method, we tested various network architectures and hyper-parameter settings. Unfortunately, we were unable to find a setting that will yield an appropriate clustering on any of the datasets for DCN, VaDE and DEPICT. IMSAT worked on two out of the five datasets, however failed to yield an appropriate clustering in fairly simple cases. Plots with typical of each of the methods on each of the five 2D datasets is shown in FIG4.To further investigate why these methods fail, we performed a sequence of experiments with the two nested'C's data, while changing the distance between the two clusters. The are shown in FIG5. We can see that all three methods fail to cluster the points correctly once the clusters cannot be linearly separated. Interestingly, although the target distribution of DEPICT was initialized with agglomerative clustering, which successfully clusters the nested'C's, its target distribution becomes corrupted throughout the training, although its loss is considerably reduced, see FIG6. DISPLAYFORM1 To prove Theorem 4.1, we begin with the following definition and lemmas. DISPLAYFORM0, where V has an even number of vertices and has a balanced partition V = S∪T, |S| = |T |, and W is an affinity matrix so that:• For any v i, v j ∈ S (resp. T), there is a path v i = v k1, v k2,..., v k l = v j ∈ S, so that for every two consecutive points v k l, v k l+1 along the path, W k l,k l+1 ≥ α.• For any v i ∈ S, v j ∈ T, W i,j ≤ β. Lemma C.2. For any integer m > 0 there exists a setX = {x 1, . . ., DISPLAYFORM1, so that for any binary partitionX =S ∪T, we can construct a set X of n = 10m points,X ⊂ X, and a balanced binary partition X = S ∪ T, |S| = |T | of it, such that DISPLAYFORM2 • For any x i, x j ∈ S (resp. T), there is a path x i, x k1, x k2,..., x k l, x j ∈ S, so that for every two consecutive points x k l, x k l+1 along the path, DISPLAYFORM3 • For any x i ∈ S, x j ∈ T, x i − x j | ≥ 1(property b).Proof. We will prove this for the case d = 3; the proof holds for any d ≥ 3.Let m > 0 be integer. We choose the setX to lie in a 2-dimensional unit grid inside a square of minimal diameter, which is placed in the Z = 0 plane. Each point x i is at a distance 1 from its neighbors. Next, given a partition of x 1,..., x m to two subsets,S andT, we will construct a set X ⊃X with n = 10m points and a partition S ∪ T that satisfy the conditions of the lemma (an illustration can be seen in Figure 7). First, we add points to obtain a balanced partition. We do so by adding m new points x m+1,..., x 2m, assigning each of them arbitrarily to eitherS orT until |S| = |T | = m. We Figure 7: Illustration of the construction of Lemma C.2. We select the setX to lie in a grid in the Z = 0 plane. Given an arbitrary dichotomyX =S ∪T (points are marked with filled circles, colored respectively in red and blue), we first add points to make the sets balanced (not shown). Next, we make a copy for S at Z = 1 and for T at Z = −1 (filled squares). We then add midpoints between each point and its copy (empty circles), and finally add more points along the minimal length spanning tree (empty squares). Together, all the red points form the set S; the blue points form the set T, and X = S ∪ T.place all these points also on grid points in the Z = 0 plane so that all 2m points lie inside a square of minimal diameter. We further add all the points inS to S and those inT to T.In the next step, we prepare a copy of theS-points at Z = 1 (with the same X, Y coordinates) and a copy of theT -points at Z = −1. We denote these copies by x 1,..., x 2m and refer to the lifted points at Z = 1 by S and at Z = −1 by T. Next, we will add 6m more points to make the full set of n = 10m points satisfy properties a and b. First, we will add the midpoint between every point and its copy, i.e., x i = (x i + x i)/2. We assign each such midpoint to S (resp. T) if it is placed between x i ∈ S and x i ∈ S (resp. T and T). Then we connect the points in S (resp. T) by a minimal length spanning tree and add 4m more points along the edges of these two spanning trees so that the added points are equally spaced along every edge. We assign the new points on the spanning tree of S to S and of T to T.We argue that the obtained point set X of size 10m satisfies the conditions of the lemma. Clearly, S ⊂ S andT ⊂ T. To show that property a is satisfied, note that the length of each spanning tree cannot exceed 2m, since the full 2m grid pointsX can be connected with a tree of length 2m − 1. It is evident therefore that every two points x i, x j ∈ S (resp. T) are connected by a path in which the distance between each two consecutive points is strictly less than 1 (property a). Property b too is satisfied because the grid points inX are at least distance 1 apart; each midpoint x i is distance 1/2 from x i and x i (and they all belong to the same set, either S or T), but its distance to the rest of the points inX exceeds 1, and the rest of the points in S (resp. T) are on the Z = 1 (resp. Z = 1) plane, and so they are at least distance 1 away from members of the opposite set which all lie in the Z ≤ 0 (resp. Z ≥ 0) half space. Lemma C.3.. Let f (·) be the spectral clustering loss DISPLAYFORM4 Let G = (X, W) be a (α,β)-separated graph, such that |X| = n ≥ 4. Let y * be a minimizer of f (y) w.r.t W, subject to 1 T y = 0, y = 1. Let ∆ S = max{y * i − y * j : x i, x j ∈ S}, and similarly ∆ T = max{y * i − y * j : x i, x j ∈ T}. Let ∆ = max {∆ S, ∆ T}. For k-means we used Python's sklearn.cluster; we used the default configuration (in particular, 300 iterations of the algorithm, 10 restarts from different centroid seeds, final are from the run with the best objective). To perform spectral clustering, we computed an affinity matrix W using, with the number of neighbors set to 25 and the scale σ set to the median distance from each point to its 25th neighbor. Once W was computed, we took the k eigenvectors of D − W corresponding to the smallest eigenvalues, and then applied k-means to that embedding. The k-means configuration was as above. In our experiments, the loss was computed with a factor of 1 m rather than 1 m 2, for numerical stability. The architectures of the Siamese net and SpectralNet are described in TAB2. Additional technical details are shown in TAB3.The learning rate policy for all nets was determined by monitoring the loss on a validation set (a random subset of the training set); once the validation loss did not improve for a specified number of epochs (see patience epochs in TAB3), we divided the learning rate by 10 (see LR decay in TAB3). Training stopped once the learning rate reached 10 −8. Typical training took about 100 epochs for a Siamese net and less than 20,000 parameter updates for SpectralNet, on both MNIST and Reuters. In the MNIST experiments, the training set for the Siamese was obtained by pairing each data point with its two nearest neighbors (in Euclidean distance). During the training of the spectral map, we construct the batch affinity matrix W by connecting each point to its nearest two neighbors in the Siamese distance. The scale σ in was set to the median of the distances from each point to its nearest neighbor. In the Reuters experiment, we obtained the training set for the Siamese net by pairing each point from that set to a random point from its 100 nearest neighbors, found by approximate nearest neighbor algorithm 4. To evaluate the generalization performance, the Siamese nets were trained using training data only. The scale σ in was set globally to the median (across all points in the dataset) distance from any point to its 10th neighbor. Finally, we used the validation loss to determine the hyper-parameters. To demonstrate that indeed the validation loss is correlated to clustering accuracy, we conducted a series of experiments with the MNIST dataset, where we varied the net architectures and learning rate policies; the Siamese net and Gaussian scale parameter σ were held fixed throughout all experiments. In each experiment, we measured the loss on a validation set and the clustering accuracy (over the entire data). The correlation between loss and accuracy across these experiments was -0.771. This implies that hyper-parameter setting for the spectral map learning can be chosen based on the validation loss, and a setup that yields a smaller validation loss should be preferred. We remark that we also use the convergence of the validation loss to determine our learning rate schedule and stopping criterion.

represent the short summary for finding the corresponding scientific article (long)

Unsupervised spectral clustering using deep neural networks

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose Janossy pooling, a method for learning deep permutation invariant functions designed to exploit relationships within the input sequence and tractable inference strategies such as a stochastic optimization procedure we call piSGD