Question: represent the text to find a very short summary (1-2 sentences max).

Question: Link prediction in simple graphs is a fundamental problem in which new links between nodes are predicted based on the observed structure of the graph. However, in many real-world applications, there is a need to model relationships among nodes which go beyond pairwise associations. For example, in a chemical reaction, relationship among the reactants and products is inherently higher-order. Additionally, there is need to represent the direction from reactants to products. Hypergraphs provide a natural way to represent such complex higher-order relationships. Even though Graph Convolutional Networks (GCN) have recently emerged as a powerful deep learning-based approach for link prediction over simple graphs, their suitability for link prediction in hypergraphs is unexplored -- we fill this gap in this paper and propose Neural Hyperlink Predictor (NHP). NHP adapts GCNs for link prediction in hypergraphs. We propose two variants of NHP --NHP-U and NHP-D -- for link prediction over undirected and directed hypergraphs, respectively. To the best of our knowledge, NHP-D is the first method for link prediction over directed hypergraphs. Through extensive experiments on multiple real-world datasets, we show NHP's effectiveness. The problem of link prediction in graphs has numerous applications in the fields of social network analysis BID24, knowledge bases BID30, bioinformatics BID26 to name a few. However, in many real-world problems relationships go beyond pairwise associations. For example, in chemical reactions data the relationship representing a group of chemical compounds that can react is inherently higher-order and similarly, the co-authorship relationship in a citation network is higher-order etc. Hypergraphs provide a natural way to model such higher-order complex relations. Hyperlink prediction is the problem of predicting such missing higher-order relationships in a hypergraph. Besides the higher-order relationships, modeling the direction information between these relationships is also useful in many practical applications. For example, in the chemical reactions data, in addition to predicting groups of chemical compounds which form reactants and/or products, it is also important to predict the direction between reactants and products, i.e., a group of reactants react to give a group of products. Directed hypergraphs BID12 provide a way to model the direction information in hypergraphs. Similar to the undirected hypergraphs, predicting the missing hyperlinks in a directed hypergraph is also useful in practical settings. Figure 1 illustrates the difference between modeling the chemical reactions data using undirected and directed hypergraphs. Most of the previous work on hyperlink prediction BID43 focus only on undirected hypergraphs. In this work we focus both on undirected and directed hypergraphs. Recently, Graph Convolutional Networks (GCNs) BID21 have emerged as a powerful tool for representation learning on graphs. GCNs have also been successfully applied for link prediction on normal graphs BID34 BID20. Inspired by the success of GCNs for link prediction in graphs and deep learning in general BID39, we propose a GCN-based framework for hyperlink prediction which works for both undirected and directed hypergraphs. We make the following contributions:Figure 1: Illustrating the difference between modeling chemical reactions data using undirected and directed hypergraphs. To the left is the undirected hypergraph, in which both the reactants and products are present in the same hyperlink. Whereas in the directed hypergraph (to the right), for a given reaction, the reactants are connected by one hyperlink and products are connected by another hyperlink and both these hyperlinks are connected by a directed link.• We propose a Graph Convolutional Networks (GCN)-based framework called Neural Hyperlink Predictor (NHP) for the problem of hyperlink prediction. To the best of our knowledge, this is the first ever deep learning based approach for this problem.• We extend the proposed NHP for the problem of hyperlink prediction in directed hypergraphs. To the best of our knowledge, this is the first ever attempt at the problem of link prediction in directed hypergraphs.• Through extensive experiments on multiple real-world datasets, we show the effectiveness of proposed NHP for link prediction in both undirected and directed hypergraphs. We have released NHP's source code at this anonymous location: https://anonymous.4open. science/repository/7d86231e-f6ba-4795-ae51-ac28d89f1521/. In this section, we briefly review related work in deep learning on graphs and link prediction on hypergraphs. Learning representations on graphs: The key advancements in learning low-dimensional node representations in graphs include matrix factorisation-based methods, random-walk based algorithms, and deep learning on graphs BID16. Our work is based on deep learning on graphs. Geometric deep learning is an umbrella phrase for emerging techniques attempting to generalise (structured) deep neural network models to non-Euclidean domains such as graphs and manifolds. The earliest attempts to generalise neural networks to graphs embed each node in an Euclidean space with a recurrent neural network (RNN) and use those embeddings as features for classification or regression of nodes or graphs BID14 BID32.A CNN-like deep neural neural network on graphs was later formulated in the spectral domain in a pioneering work BID5 by a mathematically sound definition of convolution on graph employing the analogy between the classical Fourier transforms and projections onto the eigen basis of the graph Laplacian operator BID17. Initial works proposed to learn smooth spectral multipliers of the graph Laplacian, although at high computational cost BID5 BID19. To resolve the computational bottleneck and avoid the expensive computation of eigenvectors, the ChebNet framework BID7 learns Chebyshev polynomials of the graph Laplacian (hence the name ChebNet). The graph convolutional network (GCN) BID21 ) is a simplified ChebNet framework that uses simple filters operating on 1-hop local neighborhoods of the graph. A second formulation of convolution on graph is in the spatial domain (or equivalently in the vertex domain) where the localisation property is provided by construction. One of the first formulations of a spatial CNN-like neural network on graph generalised standard molecular feature extraction methods based on circular fingerprints BID8. Subsequently, all of the above types (RNN, spectral CNN, spatial CNN on graph) were unified into a single message passing neural network (MPNN) framework and a variant of MPNN has been shown to achieve state-of-the-art on an important molecular property prediction benchmark. The reader is referred to a comprehensive literature review ) and a survey BID16 on the topic of deep learning on graphs and learning representation on graphs respectively. Below, we give an overview of related research in link prediction on hypergraphs where relationships go beyond pairwise. Link Prediction on hypergraphs: Machine learning on hypergraphs was introduced in a seminal work BID43 that generalised the powerful methodology of spectral clustering to hypergraphs and further inspired algorithms for hypergraph embedding and semi-supervised classification of hypernodes. Link prediction on hypergraph (hyperlink prediction) has been especially popular for social networks to predict higher-order links such as a user releases a tweet containing a hashtag BID22 and to predict metadata information such as tags, groups, labels, users for entities (images from Flickr) BID1. Techniques for hyperlink prediction on social networks include ranking for link proximity information BID22 and matrix completion on the (incomplete) incidence matrix of the hypergraph BID1 BID27. Hyperlink prediction has also been helpful to predict multi-actor collaborations BID35.In other works, a dual hypergraph has been constructed from the intial (primal) hypergraph to cast the hyperlink prediction as an instance of vertex classification problem on the dual hypergraph BID25. Coordinated matrix maximisation (CMM) predicts hyperlinks in the adjacency space with non-negative matrix factorisation and least square matching performed alternately in the vertex adjacency space. CMM uses expectation maximisation algorithm for optimisation for hyperlink prediction tasks such as predicting missing reactions of organisms' metabolic networks. Undirected hypergraph is an ordered pair H = (V, E) where V = {v 1, · · ·, v n} is a set of n hypernodes and E = {e 1, · · ·, e m} ⊆ 2 V is a set of m hyperlinks. The problem of link prediction in an incomplete undirected hypergraph H involves predicting missing hyperlinks fromĒ = 2 V − E based on the current set of observed hyperlinks E. The number of hypernodes in any given hyperlink e ∈ E can be any integer between 1 and 2 n. This variable cardinality of a hyperlink makes traditional graph-based link prediction methods infeasible because they are based on exactly two input features (those of the two nodes potentially forming a link). The variable cardinality problem also in an exponentially large inference space because the total number of potential hyperlinks is O(2 n). However, in practical cases, there is no need to consider all the hyperlinks inĒ as most of them can be easily filtered out. For example, for the task of finding missing metabolic reactions, we can restrict hyperlink prediction to all feasible reactions because the infeasible reactions seldom have biological meanings. In other cases such as predicting multi-author collaborations of academic/technical papers, hyperlinks have cardinalities less than a small number, as papers seldom have more than 6 authors. The number of restricted hyperlinks in such practical cases is not exponential and hence hyperlink prediction on the restricted set of hyperlinks becomes a feasible problem. Formally, a hyperlink prediction problem is a tuple (H, E), where H = (V, E) is a given incomplete hypergraph and E is a set of (restricted) candidate hyperlinks with E ⊆ E. The problem is to find the most likely hyperlinks missing in H from the set of hyperlinks E − E.Directed hypergraph BID12 is an ordered pair H = (V, E) where V = {v 1, · · ·, v n} is a set of n hypernodes and DISPLAYFORM0 V is a set of m directed hyperlinks. Each e ∈ E is denoted by (t, h) where t ⊆ V is the tail and h ⊆ V is the head with t = Φ, h = Φ. As shown in figure 1, chemical reactions can be modeled by directed hyperlinks with chemical substances forming the set V. Observe that this model captures and is general enough to subsume previous graph models:• an undirected hyperlink is the special case when t = h• a directed simple link (edge) is the special case when |t| = |h| = 1 Similar to the undirected case, the directed hyperlink prediction problem is a tuple (H, E), where H = (V, E) is a given incomplete directed hypergraph and E is a set of candidate hyperlinks with E ⊆ E. The problem is to find the most likely hyperlinks missing in H from the set of hyperlinks E − E. In this section we discuss the proposed approach for link prediction in hypergraphs. Our proposed NHP can predict both undirected and directed hyperlinks in a hypergraph. We start with how NHP can be used to predict undirected hyperlinks and then explain how NHP can be extended to the directed case. Given an undirected hypergraph H = (V, E), as a first step NHP constructs a dual hypergraph DISPLAYFORM0 is obtained by taking V * = E as the set of hypernodes and E * = {e * 1, . . ., e * n} such that e * i = {e ∈ E : v i ∈ e} with e * i corresponding to v i for i = 1,..., n. The vertex-hyperedge incidence matrices of H and H * are transposes of each other. The problem of link prediction in H can be posed as a binary node classification problem in H * BID25. A label +1 on a node in H * indicates the presence of a hyperlink in H and a label of −1 indicates the absence. For the problem of semi-supervised node classification on the dual hypergraph H *, we use Graph Convolutional Networks (GCN) on the graph obtained from the clique expansion of H *. Clique expansion BID43 BID11 ) is a standard and a simple way of approximating a hypergraph into a planar graph by replacing every hyperlink of size s with an s-clique .Graph Convolutional Network Let G = (V, E), with N = |V|, be a simple undirected graph with the adjacency matrix A ∈ R N ×N, and let the data matrix be X ∈ R N ×p. The data matrix has p-dimensional real-valued vector representations for each node in the graph. The forward model for a simple two-layer GCN takes the following simple form: DISPLAYFORM1 whereĀ =D DISPLAYFORM2 is an input-to-hidden weight matrix for a hidden layer with h hidden units and Θ ∈ R h×r is a hidden-to-output weight matrix. The softmax activation function is defined as softmax(x i) = exp(xi) j exp(xj) and applied row-wise. For semi-supervised multi-class classification with q classes, we minimise the cross-entropy error over the set of labeled examples, V L. DISPLAYFORM3 The weights of the graph convolutional network, viz. Θ and Θ, are trained using gradient descent. We note that BID25 also follow a similar approach of constructing dual graph followed by node classification for hyperlink prediction. However, ours is a deep learning based approach and BID25 do not perform link prediction in the directed hypergraphs. Figure 2: (best seen in colour) The proposed NHP framework. We convert the hypergraph with the observed hyperlinks and the candidate hyperlinks into its dual in which hyperlinks are converted into hypernodes. We then use a technique from positive unlaballed learning to get plausible negatively labelled hypernodes. The clique expansion of the hypergraph is used to approximate the hypergraph. Then a GCN is run on the graph to classify the unlabelled hypernodes. A label of +1 on e 1 indicates presence of the e 1 in the primal. For more details, refer to section 4.Learning on hypergraphs in the positive unlabelled setting The cross-entropy objective of GCN in equation 2 inherently assumes the presence of labeled data from at least two different classes and hence cannot be directly used for the positive unlabeled setting. A challenging variant of semi-supervised learning is positive-unlabelled learning BID9 which arises in many real-world applications such as text classification, recommender systems, etc. In this setting, only a limited amount of positive examples are available and the rest are unlabelled examples. In positive unlabelled learning framework, we construct a plausible negative sampling set based on data similarity BID18. We calculate the average similarity of each unlabelled point u ∈ E − E, to the positive labelled examples in E: DISPLAYFORM4 where S represents a similarity function and f represents a map that maps an example to a ddimensional emebedding space f: E → R d. We then rank the unlabelled training examples in E − E in the ascending order of their average similarity scores. We then select the top ones in the ascending order (i.e. with the lowest similarity values) to construct the set of plausible negative examples F ⊆ E − E. The intuition here is that the set of plausible negative examples contain examples most dissimilar to the positive examples. The GCN on the hypergraph can subsequently be run by minimising the objective 2 over the positive examples in E and the plausible negative examples in F i.e. V L = E ∪ F. As explained in Section 3, hyperlink prediction in directed hypergraphs is the problem of predicting directing links which are tail-head set pairs. However, in practice the collection of the tail and head sets is also incomplete. Therefore, the problem of link prediction in directed hypergraphs also requires to predict missing tail and head sets of nodes besides predicting directed links among them. The tail and head sets of nodes can be thought of as undirected hyperlinks and the directed hyperlink is between these undirected hyperlinks. A straight forward approach for this problem would be to predict the undirected hyperlinks first and then followed by predicting the directed links between pairs. However, this sequential approach may not produce desirable as the error during the training for directed part would not have any impact on the training for the undirected part. Therefore, we propose the following joint learning scheme, DISPLAYFORM0 DISPLAYFORM1 The tail hyperlink and the corresponfding head hyperlink are separate hypernodes in the dual and form directed hyperlinks in the primal (with the direction from t to h). The set W + L consists of directed hyperlinks that currently exist in the given directed hypergraph. Note that, for loss L u, the set of positively labelled hypernodes will be, DISPLAYFORM2 We sample |W + L | = |E| hypernodes (in the dual) from the unlabelled data using the positive unlabelled approach of 3 to get the set of W − L pairs. We label these pairs negative i.e. d ij1 = 0 and DISPLAYFORM3 L are used to minimise the objective 6. To explain how D is computed, we rewrite equation 1 as: DISPLAYFORM4 We use D = g(x 1, x 2) with g being a function that takes the dual hypernode representations x 1 ∈ X and x 2 ∈ X and is parameterised by for example a simple neural network. In the experiments, we used a simple 2-layer multilayer perceptron on the concatenated embeddings x 1 ||x 2 i.e.g(x 1, x 2) = MLP Θ x 1 ||x 2.We train DISPLAYFORM5, and Θ end-to-end using backpropagation. In this section, we evaluate NHP on hyperlink prediction in undirected hypergraphs. We performed two different sets of experiments whose motivations and setups are as follows. Predicting reactions of metabolic networks: Reconstructed metabolic networks are important tools for understanding the metabolic basis of human diseases, increasing the yield of biologically engineered systems, and discovering novel drug targets. We used four datasets of and we show the statistics of the datasets used in table 1. For each dataset, we randomly generated fake reactions according to the substance distribution of the existing reactions. So, the candidate reactions contain already existing ones and the randomly generated fake ones. The number of fake reactions generated is equal to the number of already existing ones. Given a small number of reactions the task is to predict the other reactions. Table 3: mean AUC (higher is better) over 10 trials. NHP achieves consistently superior performance over its baselines for all the datasets. Refer to section 5 for more details. dataset iAF692 iHN637 iAF1260b iJO1366 CORA DBLP SHC BID43 248 ± 6 289 ± 4 1025 ± 4 1104 ± 19 1056 ± 14 845 ± 18 node2vec 299 ± 10 303 ± 4 1100 ± 13 1221 ± 21 1369 ± 15 813 ± 9 CMM 170 ± 6 225 ± 10 827 ± 1 963 ± 15 1452 ± 13 651 ± 20 GCN on star expansion BID40 174 ± 5 219 ± 12 649 ± 10 568 ± 18 1003 ± 14 646 ± 15 NHP-U (ours) 313 ± 6 360 ± 5 1258 ± 9 1381 ± 9 1476 ± 20 866 ± 15 Table 4: mean (± std) number of hyperlinks recovered over 10 trials (higher is better) among the top ranked |∆E| hyperlinks. NHP achieves consistently superior performance over its baselines for all the datasets. Refer to section 5 for more details. Predicting multi-author collaborations in coauthorship networks: Research collaborations in scientific community have been extensively studied to understand team dynamics in social networks BID29 BID3 BID2 BID6. Coauthorship data provide a means to analyse research collaborations. We used cora and dblp for coauthorship data. The statistics are shown in table 2 and the construction of the datasets is pushed to the supplementary. A coauthorship hypergraph (primal) contains each author as a hypernode and each paper represents a hyperlink connecting all the authors of the paper. The corresponding dual hypergraph considers each author as a hyperlink connecting all the papers (hypernodes) coauthored by the author. The hyperlink prediction problem is given a small number of collaborations, to essentially predict other collaborations among a given set of candidate collaborations. For each dataset we randomly sampled 10% of the hypernodes to get E and then we sampled an equal number of negatively-labelled hypernodes from E − E in the positive-unlabelled setting of 3. To get the feature matrix X, we used random 32−dimensional Gaussian features (p = 32 in equation 1) for the metabolic networks and bag-of-word features shown in table 2 for the coauthorship datasets. For fake papers we generated random Gaussian bag-of-word features. We used node2vec to learn low dimensional embedding mapping, f: E → R d with d = 128. We used the clique expansion of the dual hypergraph as input graph to node2vec and cosine similarity to compute similarity between two embeddings. We compared NHP against the following state-of-the-art baselines for the same E as constructed above.• Spectral hypergraph Clustering (SHC) BID43: SHC outputs classification scores by f = (I − ξΘ) −1 y. We used SHC on the dual hypergraph.• node2vec : One of the most popular node embedding approaches. We note that have shown node2vec to be superior to DeepWalk BID31 and LINE BID36 and hence we compared against only node2vec. We used node2vec to embed the nodes of the clique expansion (of the dual). We then used an MLP on the embeddings with the semi-supervised objective of equation 2 in the positive unlabelled setting of equation 3.• Co-ordinated Matrix Maximisation (CMM): The matrix factorisation-based CMM technique uses the EM algorithm to determine the presence or absence of candidate hyperlinks.• GCN on star expansion BID40: PinSage BID40 ) is a GCN-based method designed to work on the (web-scale) bipartite graph of Pinterest. The Pinterest graph can be seen as the star expansion of a hypergraph BID0 with pins (hypernodes) on one side of the partition and boards (hyperlinks) on the other side. We have compared NHP against star expansion. This is essentially approximating the hypergraph with its star exapansion and then running a GCN over it (instead of the clique expansion of NHP). Similar to, we report mean AUC over 10 trials in table 3 and the mean number of hyperlinks recovered over 10 trials in the top ranked |∆E| ones in table 4. Note that ∆E ⊂ E is the set of missing hyperlinks with |∆E| = 0.9 * |E|. As we can observe, we consistently outperform the baselines in both the metrics. We believe this is because of the powerful non-linear feature extraction capability of GCNs. We also report Recall@ |∆E| for NHP-U for all datasets (to make the numbers across datasets somewhat comparable). It is got through dividing the mean number of hyperlinks recovered by ∆E. Table 6: mean AUC over 10 trials for all the datasets. Both the proposed models achieve similar . Refer to section 6 for more details. dataset iAF692 iHN637 iAF1260b iJO1366 node2vec + MLP 255 ± 5 237 ± 5 838 ± 13 902 ± 11 CMM + MLP 253 ± 9 241 ± 11 757 ± 26 848 ± 21 GCN on star expansion BID40 Table 7: mean (± std) number of hyperlinks recovered over 10 trials (higher is better) among the top ranked |∆E| hyperlinks. Both the proposed models achieve similar . Refer to section 6 for more details. We used the same four metabolic networks to construct directed hyperlinks. The metabolic reactions are encoded by stoichiometric matrices. The negative entries in a stoichiometric matrix indicate reactants and positive entries indicate products. We extracted only those reactions which have at least two substances in each of the reactant side and the product side and the statistics are shown in table 5. We labelled randomly sampled 10% of the hyperlinks in the data and use the remaining 90% unlabelled data for testing. Tables 6 and 7 show the on the datasets. NHP-D (joint) is the model proposed in 4.2. On the other hand, NHP-D (sequential) is the model which treats undirected hyperlink prediction and direction prediction separately. NHP-D (sequential) first runs a GCN on the clique expansion of the undirected hypergraph to get the node embeddings (without softmax) and then runs a multi-layer perceptron on the concatenated emebeddings to predict the directions. Table 9: mean (± std) number of hyperlinks recovered over 10 trials among the top ranked |∆E| hyperlinks. Positive unlabeled learning of 3 achieves consistently lower standard deviations than the the other two. The standard deviations of random negative sampling are on the higher side. Refer to section 7 for more details. We compared against the following baselines with a multi-layer perceptron (MLP):• node2vec + MLP: We used node2vec for the undirected hyperlink prediction part (as explained in section 5 and a 2-layer perceptron to predict the direction between hyperlinks with the joint objective of equation 4.• Co-ordinated Matrix Maximisation (CMM) + MLP: The matrix factorisation-based CMM technique uses the EM algorithm to determine the presence or absence of candidate hyperlinks for the following optimisation problem: DISPLAYFORM0 We used a 2−layer perceptron in a sequential manner. To get a representation for a hyperlink e, we took the mean of the representations of all the hypernodes v ∈ e from the matrix W above. Note CMM works on the primal hypergraph.• GCN on star expansion BID40 + MLP: We used the star expansion to approximate the input hypergraph and then run a GCN on it (instead of the clique expansion of NHP). A 2-layer perceptron is used to predict the direction between hyperlinks with the joint objective of equation 4.As we see in the tables, both NHP-D (joint) and NHP-D (sequential) perform similarly. This can be attributed to the fact that training data to predict directions between hyperlinks is sparse and hence the learned hypernode representations of both the models are similar. Please note that existing apporaches for link prediction on directed simple graphs cannot be trivially adopted for this problem because of the sparsity in the training data. Comparison to baselines NHP-D outperforms the baselines on 3 out of 4 datasets. The dataset iHN637 seems to be a very challenging dataset on which each model recovers less than half the number of missing hyperlinks. In order to justify positive-unlabeled learning of equation 3, we compared NHP-U and negative samples chosen uniformly at random. The for all the undirected hypergraph datasets are shown in tables 8 and 9. In the tables, we have called negative samples chosen uniformly at random from E − E as random negative sampling. We have called negative samples chosen through positive unlabeled learning of equation 3, i.e. NHP-U, as positive-unlabeled learning. Note that the numbers corresponding to this row are the same as those in tables 3 and 4.In addition to the above two, we used positive unlabeled learning technique of equation 3 to sort the hyperedges (primal) in nondecreasing order of their similarities and then selected uniformly at random from only the first half of the sorted order (i.e. most dissimilar hyperedges). We have called this technique mixed as, intuitively, it provides benefits of both positive unlabeled learning and uniform random negative sampling. More principled approaches than mixed is left for future work. As we can see in table 9, the standard deviations of random negative sampling are on the higher side. This is expected as the particular choice made for negative samples decides the decision boundary for the binary classifier. The superior AUC values of mixed in table 8 supports our intuition that it provides benefits of both positive unlabeled learning and uniform random negative sampling. The standard deviations of mixed are much lower but still higher than positive-unlabeled learning. In general, summarising the for all datasets, we believe that positive-unlabeled learning is superior to random negative sampling because of the higher confidence (low standard deviation) predictions. We have introduced NHP, a novel neural approach for hyperlink prediction in both undirected and directed hypergraphs. To the best of our knowledge, this is the first neural method for hyperlink prediction in undirected hypergraphs. NHP is also the first method for hyperlink prediction in directed hypergraphs. Through extensive experiments on multiple real-world datasets, we have demonstrated NHP's effectiveness over state-of-the art baselines. Approaches that augment GCNs with attention BID38, self-training and co-training with random walks BID23, edge-feature learning in a dual-primal setup BID28 have been recently proposed on graph-based semi-supervised learning tasks. Our NHP framework provides the flexibility to incorporate these approaches for more improved performance. An interesting future direction is predicting hyperlinks in partial-order hypergraphs . We leave extending NHP framework to inductive settings as part of future work.hyperparameter value number of hidden units 16 number of hidden layers 2 dropout rate 0.5 L2 regularisation 5 × 10 −4 learning rate 0.01 non-linearity ReLU TAB1: Hyperparameters of GCN used for all the datasets• DBLP: We used the DBLP database v4 3. We filtered out papers without abstracts, and processed each abstract by tokenizing it and removing stop-words removal. Further, we filtered out papers with one author only. This left 540532 papers. In order to ensure that the hypergraph formed would be sufficiently dense, we found the number of papers authored by each author and took the top 1000 authors as'selected authors'. Then we filtered out the papers that were not authored by at least three of the selected authors. Finally, we were left with 1590 papers by 685 of the original 1000 selected authors. To extract word features from each of these abstracts, we took all words appearing in these abstracts with a frequency greater than 50. Each abstract was thus represented by a 602-dimensional bag-of-words representation. For both datasets, we randomly sample |E| fake papers according to the author distribution of the existing non-fake papers (2708 and 1590 for CORA and DBLP respectively). We randomly generated Gaussian p dimensional features for these fake papers (1433 and 602 for CORA and DBLP respectively).

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We propose Neural Hyperlink Predictor (NHP). NHP adapts graph convolutional networks for link prediction in hypergraphs

A new method for unsupervised representation learning on graphs, relying on maximizing mutual information between local and global representations in a graph. State-of-the-art results, competitive with supervised learning.


Question: Represent the article for finding a single-sentence TLDR

Question: In this paper, we present a method for adversarial decomposition of text representation. This method can be used to decompose a representation of an input sentence into several independent vectors, where each vector is responsible for a specific aspect of the input sentence. We evaluate the proposed method on two case studies: the conversion between different social registers and diachronic language change. We show that the proposed method is capable of fine-grained con- trolled change of these aspects of the input sentence. For example, our model is capable of learning a continuous (rather than categorical) representation of the style of the sentence, in line with the reality of language use. The model uses adversarial-motivational training and includes a special motivational loss, which acts opposite to the discriminator and encourages a better decomposition. Finally, we evaluate the obtained meaning embeddings on a downstream task of para- phrase detection and show that they are significantly better than embeddings of a regular autoencoder. Despite the recent successes in using neural models for representation learning for natural language text, learning a meaningful representation of input sentences remains an open research problem. A variety of approaches, from sequence-to-sequence models that followed the work of BID37 to the more recent proposals BID2 BID29 BID8 BID25 BID36 BID5 share one common drawback. Namely, all of them encode the input sentence into just one single vector of a fixed size. One way to bypass the limitations of a single vector representation is to use an attention mechanism BID3 BID40. We propose to approach this problem differently and design a method for adversarial decomposition of the learned input representation into multiple components. Our method encodes the input sentence into several vectors, where each vector is responsible for a specific aspect of the sentence. In terms of learning different separable components of input representation, our work most closely relates to the style transfer work, which has been applied to a variety of different aspects of language, from diachronic language differences BID42 to authors' personalities BID24 and even sentiment BID17 BID13. The style transfer work effectively relies on the more classical distinction between meaning and form BID9, which accounts for the fact that multiple surface realizations are possible for the same meaning. For simplicity, we will use this terminology throughout the rest of the paper. Consider the case when we encode an input sentence into a meaning vector and a form vector. We are then able to perform a controllable change of meaning or form by a simple change applied to these vectors. For example, we can encode two sentences written in two different styles, then swap the form vectors while leaving the meaning vectors intact. We can then generate new unique sentences with the original meaning, but written in a different style. In the present work, we propose a novel model for this type of decomposition based on adversarialmotivational training and design an architecture inspired by the GANs BID14 and adversarial autoencoders BID26. In addition to the adversarial loss, we use a special motivator BID0, which, in contrast to the discriminator, is used to provide a motivational loss to encourage the model to better decomposition of the meaning and the form, as well as specific aspects of meaning. We make all the code publicly available on GitHub 1.We evaluate the proposed methods for learning separate aspects of input representation on the following case studies:1. Learning to separate out a representation of the specific diachronic slice of language. One may express the same meaning using the Early Modern English (e.g. What would she have?) and the contemporary English (What does she want?)2. Learning a representation for a social register BID16 -that is, subsets of language appropriate in a given context or characteristic of a certain group of speakers. These include formal and informal language, the language used in different genres (e.g., fiction vs. newspapers vs. academic texts), different dialects, and even literary idiostyles. We experiment with the registers corresponding to the titles of scientific papers vs. newspaper articles. As mentioned above, the most relevant previous work comes from the style transfer research, and it can be divided into two groups:1. Approaches that aim to generate text in a given form. For example, the task may be to produce just any verse as long as it is in the "style" of the target poet.2. Approaches that aim to induce a change in either the "form" or the "meaning" of an existing utterance. For example, "Good bye, Mr. Anderson." can be transformed to "Fare you well, good Master Anderson" BID42 ).An example of the first group is the work by BID31, who trained several separate networks on verses by different hip-hip artists. An LSTM network successfully generated verses that were stylistically similar to the verses of the target artist (as measured by cosine distance on TfIdf vectors). More complicated approaches use language models that are conditioned in some way. For example, BID24 produced product reviews with a target rating by passing the rating as an additional input at each timestep of an LSTM model. BID38 generated reviews not only with a given rating but also for a specific product. At each timestep a special context vector was provided as input, gated so as to enable the model to decide how much attention to pay to that vector and the current hidden state. BID23 used "speaker" vectors as an additional input to a conversational model, improving consistency of dialog responses. Finally, BID12 performed an extensive evaluation of conditioned language models based on "content" (theme and sentiment) and "style" (professional, personal, length, descriptiveness). Importantly, they showed that it is possible to control both "content" and "style" simultaneously. Work from the second group can further be divided into two clusters by the nature of the training data: parallel aligned corpora, or non-aligned datasets. The aligned corpora enable approaching the problem of form shift as a paraphrasing or machine translation problem. BID42 used statistical and dictionary-based systems on a dataset of original plays by Shakespeare and their contemporary translations. BID4 trained an LSTM network on 33 versions of the Bible. BID19 used a Pointer Network BID41, an architecture that was successfully applied to a wide variety of tasks BID27 BID15 BID32, to enable direct copying of the input tokens to the output. Note that these works use BLEU BID30 as the main, or even the only evaluation measure. This is only possible in cases where a parallel corpus is available. Recently, new approaches that do not require a parallel corpora were developed in both CV and NLP. BID17 succeeded in changing tense and sentiment of sentences with a two steps procedure based on a variational auto-encoder (VAE) BID21. After training a VAE, a discriminator and a generator are trained in an alternate manner, where the discriminator tries to correctly classify the target sentence attributes. A special loss component forces the hidden representation of the encoded sentence to not have any information about the target sentence attributes. BID28 used a VAE to produce a hidden representation of a sentence, and then modify it to match the desired form. Unlike BID17, they do not separate the form and meaning embeddings. BID34 applied a GAN to align the hidden representation of sentences from two corpora and force them to do not have any information about the form via adversarial loss. During the decoding, similarly the work by BID24, special "style" vectors are passed to the decoder at every timestep to produce a sentence with the desired properties. The model is trained using the Professor-Forcing algorithm BID22. BID20 worked directly on hidden space vectors that are constrained with the same adversarial loss instead of outputs of the generator, and use two different generators for two different "styles". Finally, BID13 proposed two models for generating sentences with the target properties using an adversarial loss, similarly to BID34 and BID20.Comparison with previous work In contrast to the proposals of BID42, BID4, BID19, our solution does not require a parallel corpus. Furthermore, unlike the model by BID34, our model works directly on representation of sentences in the hidden space. Most importantly, in contrast to the proposals by BID28, BID17, BID20, BID13, our model produces a representation for both meaning and form and does not treat the form as a categorical (in the vast majority of works, binary) variable. Although the form was represented as dense vectors in previous work, it is still just a binary feature, as they use a single pre-defined vector for each form, with all sentences of the same form assigned the same form vector. In contrast, our work treats form as a truly continuous variable, where each sentence has its own, unique, form vector. Treating meaning and form not as binary/categorical, but as continuous is more consistent with the reality of language use, since there are different degrees of overlap between the language used by different registers or in different diachronic slices. Indeed, language change is gradual, and the acceptability of expressions in a given register also forms a continuum, so one expects a substantial overlap between the grammar and vocabulary used, for example, on Twitter and by New York Times. To the best of our knowledge, this is the first model that considers linguistic form in the task of text generation as a continuous variable. One significant consequence of learning a continuous representation for form is that it allows the model to work with a large, and potentially infinite, number of forms. Note that in this case the locations of areas of specific forms in the vector style space would reflect the similarity between these forms. For example, the proposed model could be directly applied to the authorship attribution problem. In this case, each author would have their own area in the form space, and the more similar the authors are in terms of writing style, the closer these areas would be to each other. We performed preliminary experiments on this and report the in Appendix A. Let us formulate the problem of decomposition of text representation on an example of controlled change of linguistic form and conversion of Shakespeare plays in the original Early Modern to contemporary English. Let X a be a corpus of texts DISPLAYFORM0 and X b be a corpus of texts DISPLAYFORM1 We assume that the texts in both X a and X b has the same distribution of meaning m ∈ M. The form f, however, is different and generated from a mixture of two distributions: DISPLAYFORM2 where f a and f b are two different languages (Early Modern and contemporary English). Intuitively,we say that a sample x i has the form f a if α. The goal of dissociation meaning and form is to learn two encoders E m: X → M and E f: X → F for the meaning and form correspondingly, and the generator G: M, F → X such that ∀j ∈ {a, b}, ∀k ∈ {a, b}: DISPLAYFORM3 That is, the form of a generated sample depends exclusively on the provided f j and can be the in the same domain for two different m u and m v from two samples from different domains X a and X b.Note that, in contrast to the previously proposals, the form f is not a categorical variable but a continuous vector. This enables fine-grained controllable change of form: the original form f i is changed to reflect the form of the specific target sentence f j with its own unique α a and α b while preserving the original meaning m i.An important caveat concerns the core assumption of the similar meaning distribution in the two corpora, which is also made in all other works reviewed in Section 2. It limits the possible use of this approach to cases where the distributions are in fact similar (i.e. parallel or at least comparable corpora are available). It does not apply to many cases that could be analyzed in terms of meaning and form. For example, books for children and scholarly papers are both registers, they have their own form (i.e. specific subsets of linguistic means and structure conventions) -but there is little overlap in the content. This would make it hard even for a professional writer to turn a research paper into a fairy tale. Encoder encodes the inputs sentences into two latent vectors m and f. The Generator takes them as the input and produces the output sentence. During the training, the Discriminator is used for an adversarial loss that forces m to do not carry any information about the form, and the M otivator is used for a motivational loss that encourages f to carry the needed information about the form. Our solution is based on a widely used sequence-to-sequence framework BID37 and consists of four main parts. The encoder E encodes the inputs sequence x into two latent vectors m and f which capture the meaning and the form of the sentence correspondingly. The generator G then takes these two vectors as the input and produces a reconstruction of the original input sequencex. The encoder and generator by themselves will likely not achieve the dissociation of the meaning and form. We encourage this behavior in a way similar to Generative Adversarial Networks (GANs) BID14, which had an overwhelming success the past few years and have been proven to be a good way of enforcing a specific distribution and characteristics on the output of a model. Inspired by the work of BID0 and the principle of "carrot and stick" BID33, in contrast to the majority of work that promotes pure adversarial approach BID14 BID34 BID13, we propose two additional components, the discriminator D and the motivator M to force and motivate the model to learn the dissociation of the meaning and the form. Similarly to a regular GAN model, the adversarial discriminator D tries to classify the form f based on the latent meaning vector m, and the encoder E is penalized to make this task as hard as possible. Opposed to such vicious behaviour, the motivator M tries to classify the form based on the latent form vector f, as it should be done, and encourages the encoder E to make this task as simple as possible. We could apply the adversarial approach here as well and force the distribution of the form vectors to fit a mixture of Gaussians (in this particular case, a mixture of two Guassians) with another discriminator, as it is done by BID26, but we opted for the "dualistic" path of two complimentary forces. Both the encoder E and the generator G are modeled with a neural network. Gated Recurrent Unit (GRU) BID6 ) is used for E to encode the input sentence x into a hidden vector h = GRU(x).The vector h is then passed through two different fully connected layers to produce the latent vectors of the form and the meaning of the input sentence: DISPLAYFORM0 We use θ E to denote the parameters of the encoder E: W m, b m, W f, b f, and the parameters of the GRU unit. The generator G is also modelled with a GRU unit. The generator takes as input the meaning vector m and the form vector f, concatenates them, and passes trough a fully-connected layer to obtain a hidden vector z that represents both meaning and form of the original input sentence: DISPLAYFORM1 After that, we use a GRU unit to generate the output sentence as a probability distribution over the vocabulary tokens: DISPLAYFORM2 We use θ G to denote the parameters of the generator G: W z, b m, and the parameters of the used GRU. The encoder and generator are trained using the standard reconstruction loss: DISPLAYFORM3 The representation of the meaning m produced by the encoder E should not contain any information about the form f. We achieve this by using an adversarial approach. First, we train a discriminator D, consisting of several fully connected layers with ELU activation function BID7 between them, to predict the form f of a sentence by its meaning vector:f D = D(m), wheref is the score (logit) reflecting the probability of the sentence x to belong to one of the form domains. Motivated by the Wasserstein GAN BID1, we use the following loss function instead of the standard cross-entropy: DISPLAYFORM0 Thus, a successful discriminator will produce negative scoresf for sentences from X a and positive scores for sentences from X b. This discriminator is then used in an adversarial manner to provide a learning signal for the encoder and force dissociation of the meaning and form by maximizing L D: L adv (θ E) = −λ adv L D, where λ adv is a hyperparameter reflecting the strength of the adversarial loss. Note that this loss applies to the parameters of the encoder. Our experiments showed that it is enough to have just the discriminator D and the adversarial loss L adv to force the model to dissociate the form and the meaning. However, in order to achieve a better dissociation, we propose to use a motivator M and the corresponding motivational loss. Conceptually, this is the opposite of the adversarial loss, hence the name. As the discriminator D, the motivator M learns to classify the form f of the input sentence. However, its input is not not the meaning vector but the form vector: DISPLAYFORM0 The motivator has the same architecture as the discriminator, and the same loss function. While the adversarial loss forces the encoder E to produce a meaning vector m with no information about the form f, the motivational loss encourages E to encode this information in the form vector by minimizing DISPLAYFORM1 The overall training procedure follows the methods for training GANs BID14 BID1 and consists of two stages: training the discriminator D and the motivator M, and training the encoder E and the generator G.In contrast to BID1, we do not train the D and M more than the E and the G. In our experiments we found that simple training in two stages is enough to achieve dissociation of the meaning and the form. Encoder and generator are trained with the following loss function that combines reconstruction loss with the losses from the discriminator and the motivator: DISPLAYFORM0 Similarly to the evaluation of style transfer in CV ), evaluation of this task is difficult. We follow the approach of; BID34 and recently proposed by BID13 methods of evaluation of "transfer strength" and "content preservation". The authors showed the proposed automatic metrics to a large degree correlate with human judgment and can serve as a proxy. Below we give an overview of these metrics. Transfer Strength. The goal of this metric is to capture whether the form has been changed successfully. To do that, a classifier C is trained on the two corpora, X a and X b to recognize the linguistic "form" typical of each of them. After that, a sentence the form/meaning of which was changed is passed to the classifier. The overall accuracy reflects the degree of success of changing the form/meaning. This approach is widely used in CV, and was applied in NLP as well BID34.In our experiments we used a GRU unit followed by four fully-connected layers with ELU activation functions between them as the classifier. Content preservation Note that transfer strength by itself does not capture the overall quality of a changed sentence. A extremely overfitted model that produces the same, the most characteristic sentence of one corpus all the time would have a high score according to this metric. Thus, we need to measure how much of the meaning was preserved while changing the form. To do that, BID13 proposed to use a cosine similarity based metric using pretrained word embeddings. First, a sentence embedding is computed by concatenation of max, mean, and average pooling over the timesteps: DISPLAYFORM1 Next, the cosine similarity score s i between the embedding v s i of the original source sentence and the target sentence with the changed form v t i is computed, and the scores across the dataset are averaged to obtain the total score: DISPLAYFORM2 The metrics described above treat the form as a categorical (in most cases, even binary) variable. This was not a problem in previous work since the change of form could be done by just inverting the form vector. Our work, in contrast, treats the form as a continuous variable, and, therefore, we cannot just use the proposed metrics directly. To enable a fair comparison, we propose the following procedure. For each sentence s a s in the test set from the corpus X a we sample k = 10 random sentence from the corpus X b of the opposite form. After that, we encode them into the meaning m i and form DISPLAYFORM3 We then generate a new sentence with its original meaning vector m s and the ing form vector f avg, and use it for evalation. This process enables a fair comparison with the previous works that treat form as a binary variable. We performed an extensive evaluation of the proposed method on several dataset that reflect different changes of meaning, form, or specific aspects of meaning, such as sentiment polarity. Changing form: register This experiment is conducted with a dataset of titles of scientific papers and news articles published by BID13. This dataset (referred to as "Headlines") contains titles of scientific articles crawled from online digital libraries, such as "ACM Digital Library" and "arXiv". The titles of the news articles are taken from the "News Aggregator Data Set" from UCI Machine Learning Repository BID10 Changing form: language diachrony Diachronic language change is explored with the dataset composed by BID42. It includes the texts of 17 plays by William Shakespeare in the original Early Modern English, and their translations into contemporary English. We randomly permuted all sentences from all plays and sampled the training, validation, and test sets. Note that this is the smallest dataset in our experiments. Previous work on style transfer for text also included the experiments with changing sentiment polarity BID34 BID13. We do not report the experiments with sentiment data, since the change in sentiment polarity corresponds to a change in a specific aspect of meaning, rather than form. We therefore believe the comparison with these data would not be instructive. Probably, the most recent and similar to our work is the model proposed by BID13, in particular the "style-embedding" model. We implemented this model to provide a baseline for comparison. The classifier used in the transfer strength metric achieves very high accuracy (0.832 and 0.99 for the Shakespeare and Headlines datasets correspondingly). These concur with the of BID34 and BID13, and show that the two forms in the corpora are significantly different. Following BID13, we show the of different configuration of the size of the form and meaning vectors on FIG2. Namely, we report combinations of 64 and 256-dimensional vectors. Note that the sizes of the form vector are important. The larger is the form vector, the higher is the transfer strength, but smaller is content preservation. This is consistent with BID13, where they observed a similar behaviour. It is clear that the proposed method achieves significantly better transfer strength then the previously proposed model. It also has a lower content preservation score, which means that it repeats fewer exact words from the source sentence. Note that a low transfer strength and very high (0.9) content preservation score means that the model was not able to successfully learn to transfer the form and the target sentence is almost identical to the source sentence. The Shakespeare dataset is the hardest for the model in terms of transfer strength, probably because it is the smallest dataset, but the proposed method performs consistently well in transfer of both form and meaning and, in contrast to the baseline. Fluency of generated sentences Note that there is no guarantee that the generated sentences would be coherent after switching the form vector. In order to estimate how this switch affects the fluency of generated sentences, we trained a language model on the Shakespeare dataset and calculated the perplexity of the generated sentences using the original form vector and the average of form vectors of k random sentences from the opposite style (see subsubsection 5.1.1). While the perplexity of such sentences does go up, this change is not big (6.89 vs 9.74). To investigate the impact of the motivator, we visualized form and meaning embeddings of 1000 random samples from the Headlines dataset using t-SNE algorithm BID39 with the Multicore-TSNE library . The is presented in FIG3.There are three important observations. First, there is no clear separation in the meaning embeddings, which means that any accurate form transfer is due to the form embeddings, and the dissociation of form and meaning was successful. Second, even without the motivator the model is able to produce the form embeddings that are clustered into two group. Recall from section 4 that without the motivational loss there are no forces that influence the form embeddings, but nevertheless the model learns to separate them. However, the separation effect is much more pronounced in the presence of motivator. This explains why the motivator consistently improved transfer strength of ADNet, as shown in FIG2. 6.2 QUALITATIVE EVALUATION Table 1 and Table 2 show several examples of the successful form/meaning transfer achieved by ADNet. Table 1 presentes the of an experiment that to some extent replicates the approach taken by the authors who treat linguistic form as a binary variable BID34 BID13. The sentences the original Shakespeare plays were averaged to get the "typical" Early Modern English form vector. This averaged vector was used to decode a sentence from the modern English translation back into the original. The same was done in the opposite direction. → This man will tell us everything. (EME) I've done no more to caesar than you will do to me. (CE) → I have done no more to caesar than, you shall do to me. (EME) Table 1: Decoding of the source sentence from Early Modern English (EME) into contemporary English (CE), and vice versa. Table 2 illustrates the possibilities of ADNet on fine-grained transfer applied to the change of register. We encoded two sentences in different registers from the Headlines dataset to produce form and meaning embeddings, and then we decoded the first sentence with the meaning embedding of the second, and vice versa. As can be seen from Table 2, the model correctly captures the meaning of sentences and decodes them using the form of the source sentences. Note how the model preserves specific words and the structure of the source sentence. In particular, note how in the first example, the model decided to put the colon after the "crisis management", as the source form sentence has this syntactic structure ("A review:"). This is not possible in the previously proposed models, as they treat form as just a binary variable. A review: detection techniques for LTE system Crisis management: media practices in telecommunication management Situation management knowledge from social media A review study against intelligence internet Security flaw could not affect digital devices, experts say Semantic approach approach: current multimedia networks as modeling processes Semantic approach to event processing Security flaw to verify leaks Table 2: Flipping the meaning and the form embeddings of two sentence from different registers. Note the use of the colon in the first example, and the use of the "to"-constructions in the second example, consistent with the form of the source sentences. We conducted some experiments to test the assumption that the derived meaning embeddings should improve performance on downstream tasks that require understanding of the meaning of the sentences regardless of their form. We evaluated embeddings produced by the ADNet, trained in the Headlines dataset, on a task of paraphrase detection. We used the SentEval toolkit BID8 and the Microsoft Research Paraphrase Corpus BID11. The F1 scores on this task for different models are presented in Table 3. Note that all models, except InferSent, are unsupervised. The InferSent model was trained on a big SNLI dataset, consisting of more than 500,000 manually annotated pairs. ADNet achieves the the highest score among the unsupervised systems and outperforms the regular sequence-to-sequence autoencoder with a large gap. Table 3: F1 scores on the task of paraphrase detection using the SentEval toolkit BID8 7 In this paper, we presented ADNet, a new model that performs adversarial decomposition of text representation. In contrast to previous work, it does not require a parallel training corpus and works directly on hidden representations of sentences. Most importantly, is does not treat the form as a binary variable (as done in most previously proposed models), enabling a fine-grained change of the form of sentences or specific aspects of meaning. We evaluate ADNet on two tasks: the shift of language register and diachronic language change. Our solution achieves superior , and t-SNE visualizations of the learned meaning and style embeddings illustrate that the proposed motivational loss leads to significantly better separation of the form embeddings.

Represent the short summary for finding the corresponding scientific article (long)

A method which learns separate representations for the meaning and the form of a sentence

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.