Question: represent the article for finding a single-sentence tldr

Question: We present a Deep Neural Network with Spike Assisted Feature Extraction (SAFE-DNN) to improve robustness of classification under stochastic perturbation of inputs. The proposed network augments a DNN with unsupervised learning of low-level features using spiking neuron network (SNN) with Spike-Time-Dependent-Plasticity (STDP). The complete network learns to ignore local perturbation while performing global feature detection and classification. The experimental on CIFAR-10 and ImageNet subset demonstrate improved noise robustness for multiple DNN architectures without sacrificing accuracy on clean images. There is a growing interest in deploying DNNs in autonomous systems interacting with physical world such as autonomous vehicles and robotics. It is important that an autonomous systems make reliable classifications even with noisy data. However, in a deep convolutional neural networks (CNN) trained using stochastic gradient descent (SGD), pixel level perturbation can cause kernels to generate incorrect feature maps. Such errors can propagate through network and degrade the classification accuracy (Nazaré et al.; ). Approaches for improving robustness of a DNN to pixel perturbation can be broadly divided into two complementary categories. First, many research efforts have developed image de-noising (or filtering) networks that can pre-process an image before classification, but at the expense of additional latency in the processing pipeline (; ; ; ; ;). De-noising is an effective approach to improve accuracy under noise but can degrade accuracy for clean images . Moreover, de-noising networks trained on a certain noise type do not perform well if the a different noise structure is experienced during inference . Advanced de-noising networks are capable of generalizing to multiple levels of a type of noise and effective for different noise types (; ;). But high complexity of these network makes them less suitable for real-time applications and lightweight platforms with limited computational and memory resources. An orthogonal approach is to develop a classification network that is inherently robust to input perturbations. Example approaches include training with noisy data, introducing noise to network parameters during training, and using pixel level regularization (; Nazaré et al.;;; ). These approaches do not change the processing pipeline or increase computational and memory demand during inference. However, training-based approaches to design robust DNNs also degrade classification accuracy for clean images, and more importantly, are effective only when noise structure (and magnitude) during training and inference closely match. Therefore, a new class of DNN architecture is necessary for autonomous system that is inherently resilient to input perturbations of different type and magnitude without requiring training on noisy data, as well as computationally efficient. Towards this end, this paper proposes a new class of DNN architecture that integrates features extracted via unsupervised neuro-inspired learning and supervised training. The neuro-inspired learning, in particular, spiking neural network (SNN) with spike-timing-dependent plasticity (STDP) is an alternative and unsupervised approach to learning features in input data . However, the classification accuracy of a STDP-learned SNN for complex datasets is much lower than a that of a DNN. The fundamental premise of this paper is that, augmenting the feature space of a supervised (trained) DNN with features extracted by an SNN via STDP-based learning increases robustness of the DNN to input perturbations. We argue that stochastic gradient descent (SGD) based back-propagation in a DNN enables global learning between low-level pixel-to-pixel interactions and high-level detection and classification. On the other hand, STDP performs unsupervised local learning and extracts low-level features under spatial correlation. By integrating features from global (supervised training) and local (STDP) learning, the hybrid network "learns to ignore" locally uncorrelated perturbations (noise) in pixels while extracting the correct feature representation from the overall image. Consequently, hybridization of SGD and STDP enables robust image classification under noisy input while preserving the accuracy of the baseline DNN for clean images. We present a hybrid network architecture, referred to as Spike Assisted Feature Extraction based Deep Neural Network (SAFE-DNN), to establish the preceding premise. We develop an integrated learning/training methodology to couple the features extracted via neuro-inspired learning and supervised training. In particular, this paper makes the following contributions: • We present a SAFE-DNN architecture (Figure 1) that couples STDP-based robust learning of local features with SGD based supervised training. This is achieved by integrating a spiking convolutional module within a DNN pipeline. • We present a novel frequency-dependent stochastic STDP learning rule for the spiking convolutional demonstrating local competitive learning of low level features. The proposed learning method makes the feature extracted by the spiking convolutional module robust to local perturbations in the input image. • We develop a methodology to transform the STDP-based spiking convolution to an equivalent CNN. This is achieved by using a novel special neuron activation unit (SAU), a non-spiking activation function, that facilitates integration of the SNN extracted features within the DNN thereby creating a single fully-trainable deep network. The supervised (SGD-based) training is performed in that deep network after freezing the STDP-learnt weights in the spiking CNN module. We present implementations of SAFE-DNN based on different deep networks including MobileNet, ResNet and DenseNet (, ,) to show the versatility of our network architecture. Experiment is conducted for CIFRA10 and ImageNet subset considering different types of noise, including Gaussian, Wald, Poisson, Salt&Paper, and adversarial noise demonstrating robust classification under input noise. Unlike training-based approaches, SAFE-DNN shows improved accuracy for a wide range of noise structure and magnitude without requiring any prior knowledge of the perturbation during training and inference and does not degrade the accuracy for clean images (even shows marginal improvement in many cases). SAFE-DNN complements, and can be integrated with, de-noising networks for input pre-processing. However, unlike de-noising networks, the SAFE-DNN has negligible computation and memory overhead, and does not introduce new stages in the processing pipeline. Hence, SAFE-DNN is an attractive architecture for resource-constrained autonomous platforms with real-time processing. We note that, SAFE-DNN differs from deep SNNs that convert a pre-trained DNN to SNN (,). Such networks function as a spiking network during inference to reduce energy; however, the learning is still based on supervision and back-propagation. In contrast, SAFE-DNN hybridizes STDP and SGD during learning but creates a single hybrid network operating as a DNN during inference. Spiking neural network uses biologically plausible neuron and synapse models that can exploit temporal relationship between spiking events . There are different models that are developed to capture the firing pattern of real biological neurons. We choose to use Leaky Integrate Fire (LIF) model in this work described by: where, a, b and c are parameters that control neuron dynamics, and I is the sum of current signal from all synapses that connects to the neuron. In SNN, two neurons connected by one synapse are referred to as pre-synaptic neuron and postsynaptic neuron. Conductance of the synapse determines how strongly two neurons are connected and learning is achieved through modulating the conductance following an algorithm named spike-timingdependent-plasticity (STDP) (; ;). With two operations of STDP: long-term potentiation (LTP) and long-term depression (LTD), SNN is able to extract the causality between spikes of two connected neurons from their temporal relationship. More specifically, LTP is triggered when post-synaptic neuron spikes closely after a pre-synaptic neuron spike, indicating a causal relationship between the two events. On the other hand, when a post-synaptic neuron spikes before pre-synaptic spike arrives or without receiving a pre-synaptic spike at all, the synapse goes through LTD. For this model the magnitude of modulation is determined by : In the functions above, ∆G p is the magnitude of LTP actions, and ∆G d is the magnitude of LTD actions. α p, α d, β p, β d, G max and G min are parameters that are tuned based on specific network configurations. The gradient descent based weight update process in a DNN computes the new weight as W = W − η∇L, where the gradient of loss function L is taken with respect to weight:. Consider cross entropy loss as an example for L, weight optimization of element i is described by: Here η is the rate for gradient descent; N is the number of classes; y n is a binary indicator for the correct label of current observation andŷ n is the predicated probability of class n by the network. For equation, gradient is derived based on the output prediction probabilitiesŷ and ground truth. Such information is available only at the output layer. To generate the gradient, the output prediction (or error) has to be back-propagated from the output layer to the target layer using chain rule. Aŝ y = g(W, X) with g being the logistic function and X the input image, the prediction probabilities are the outcome of the entire network structure. Consider the low level feature extraction layers in a deep network. Equation suggests that gradient of the loss with respect to a parameter is affected by all pixels in the entire input image. In other words, the back-propagation makes weight update sensitive to non-neighboring pixels. This facilitates global learning and improve accuracy of higher level feature detection and classification. However, the global learning also makes it difficult to strongly impose local constraints during training. Hence, the network does not learn to ignore local perturbations during low-level feature extraction as it is trained to consider global impact of each pixel for accurate classifications. This means that during inference, although a noisy pixel is an outlier from the other pixels in the neighbourhood, a DNN must consider that noise as signal while extracting low-level features. The ing perturbation from pixel level noise propagates through the network, and degrades the classification accuracy. The preceding discussion suggests that, to improve robustness to stochastic input perturbation (noise), the low level feature extractors must learn to consider local spatial correlation. The local learning will allow network to more effectively "ignore" noisy pixels while computing the low-level feature maps and inhibit propagation of input noise into the DNN pipeline. The motivation behind SAFE-DNN comes from the observation that STDP in SNN enables local learning of features. Compared to conventional DNN, SNN conductance is not updated through gradient descent that depends on back propagation of global loss. Consider a network with one spiking neuron and n connected input synapses, a spiking event of the neuron at time t spike and timing of closest spikes from all input spike trains T input, the modulated conductance is given by: Here input is spike timing difference, r is the magnitude function (Equation 2) and p is the modulation probability function (Equation 5). The value of t spike is a of the neuron's response to the collective sum of input spike trains in one kernel. Hence, the modulation of weight of each synapse in a SNN depends only on other input signals within the same (local) receptive field. Moreover, as the correlation between the spike patterns of neighboring pre-synaptic neurons controls and causes the post-synaptic spike, STDP helps the network learn the expected spatial correlation between pixels in a local region. During inference, if the input image contains noise, intensity of individual pixel can be contaminated but within a close spatial proximity the correlation is better preserved. As the SNN has learned to respond to local correlation, rather than individual pixels, the neuron's activity experiences less interference from local input perturbation. In other words, the SNN "learns to ignore" local perturbations and hence, the extracted features are robust to noise. 4.1 NETWORK ARCHITECTURE Fig. 1 (a) shows an illustrative implementation of SAFE-DNN. The network contains spiking layers placed contiguously to form the spiking convolution module, along with conventional CNN layers. The spiking convolution module is placed at the front to enable robust extraction of local and low-level features. Further, to ensure that the low-level feature extraction also considers global learning, which is the hallmark of gradient backpropagation as discussed in section 3, we place several conventional CNN layers of smaller size in parallel with the spiking convolution module. This is called the auxiliary CNN module. The output feature map of the two parallel modules is maintained to have the same height and width, and concatenated along the depth to be used as input tensor to the remaining CNN layers, referred to as the main CNN module. Main CNN module is responsible for higher level feature detection as well as the final classification. The main CNN module can be designed based on existing deep learning models. The concatenation of features from auxilary CNN and spikining convolutional module helps integrate global and local learning. The first convolution layer and the following one block from the original network architecture are dropped and the remaining layers are used as the mian CNN module of SAFE-MobileNetV2. We show that SAFE-DNN is a versatile network by testing three configurations in this work, which have the main CNN module based on MobileNetV2, ResNet101 and DenseNet121, respectively. The storage and computational complexity of the networks are shown in Table 1. It can be observed that SAFE-DNN implementations do not introduce a significant overhead to the baseline networks. In the dynamical system of SNN, neurons transmit information in the form of spikes, which are temporally discrete events that spread across multiple simulation time steps. This requires input signal intensity to be converted to spike trains, and a number of time steps for neurons to respond to input stimulus. Such mechanism is different from that of the conventional DNN, which takes only one time step for data to propagate through the network. Due to this reason the native SNN model can not be used in spiking convolution module of SAFE-DNN. Two potential solutions to this problem are, running multiple time steps for every input, or, adapting the spiking convolution module to single-time-step response system. Since the first slows down both training and inference by at least one order of magnitude, we choose the latter. Training Process. We separate STDP-based learning and DNN training into two stages. In the first stage, the spiking convolution module operates in isolation and learns all images in the training set without supervision. The learning algorithm follows our novel frequency dependent STDP method described next in section 4.2. In the second stage, network parameters are first migrated to the spiking convolution module of SAFE-DNN. The network building blocks of the spiking convolutional module go through a conversion process shown in Fig. 1 (b). The input signal to spike train conversion process is dropped, and conductance matrix is re-scaled to be used in the new building block. Batch normalization is inserted after the convolution layer. In order to preserve the non-linear property of spiking neurons, a special activation unit (SAU) is designed to replace the basic spiking neuron model. Details about SAU is discussed later in section 4.3. Once the migration is completed, the entire SAFE-DNN is then trained fully using statistical method, while weights in the spiking convolution module are kept fixed to preserve features learned by SNN. Network inference is performed using the network architecture created during the second stage of training i.e. instead of the baseline LIF, the SAU is used for modeling neurons. Frequency-dependent stochastic STDP The STDP algorithm discussed in 2 captures the basic exponential dependence on timing of synaptic behavior, but does not address the associative potentiation issue in STDP (; ;). Associativity is a temporal specificity such that when a strong (in case of our SNN model, more frequent) input and a weak (less frequent) input into one neuron induce a post-synaptic spike, a following conductance modulation process is triggered equivalently for the both. In the context of STDP based SNN, associativity can cause erroneous conductance modulation if unaccounted for (She et al. (2019b) ). Therefore, we propose a frequency-dependent (FD) stochastic STDP that dynamically adjust the probability of LTP/LTD based on input signal frequency. The algorithm is described by: In this algorithm, τ d and τ p are time constant parameters. ∆t is determined by subtracting the arrival time of the pre-synaptic spike from that of the post-synaptic spike (t post − t pre). Probability of LTP P p is higher with smaller ∆t, which indicates a stronger causal relationship. The probability of LTD P d is higher when ∆t is larger. γ p and γ d controls the peak value of probabilities. f max and f min define the upper and lower limit of input spike frequency and f is the value of input spike frequency. When input spike originates from a weak input, the probability declines faster than that from a strong input. As a , pre-synaptic spike time of weak input needs to be much closer to the post-synaptic spike than that of strong input to have the same probability of inducing LTP, i.e. the window for LTP is narrower for weak input. The same rule applies to LTD behavior. As will be shown in the following section, FD stochastic STDP exhibits better learning capability than conventional STDP. The architecture of the spiking convolutional module is shown in Fig. 3. This architecture resembles conventional DNN but have some differences. First, the 8-bit pixel intensity from input images is converted to spike train with frequency over a range from f min to f max. The input spike train matrix connects to spiking neurons in the spiking convolution layer in the same way as conventional 2D convolution, which also applies for connections from one spiking convolution layer to the next. All connections as mentioned are made with plastic synapses following STDP learning rule. When a neuron in the convolution layer spikes, inhibitory signal is sent to neurons at the same (x,y) coordinate across all depth in the same layer. This cross-depth inhibition prevents all neurons at the same location from learning the same feature. Overall, such mechanism achieves a competitive local learning behavior of robust low level features that are crucial to the implementation of SAFE-DNN. A basic property of spiking neuron is that a number of spikes need to be received before a neuron reaches spiking state and emits one spike. In a two layer network this does not cause a problem but for multiple-layer network it prohibits spiking signal to travel deep down. Due to the diminishing spiking frequency of multiple-layer SNN, a layer-by-layer learning procedure is used. When the first layer completes learning, its conductance matrix is kept fixed and cross-depth inhibition disabled. Next, all neurons in the first layer are adjusted to provide higher spiking frequency by lowering the spiking threshold V th. The effect of changing V th is illustrated in Fig.4. In such way, neurons in the first layer receive input from input images and produce enough spikes that can facilitate learning behavior of the second layer. The same process is repeated until all layers complete learning. Consider the spike conversion process of SNN, given an input value of X ∈ 0, 1 and input perturbation ξ, conversion to spike frequency with range ∈ f min, f max is applied such that F = Clip {(X + ξ)(f max − f min)}. For the duration of input signal T input, the total received spikes for the recipient is N spike = F * T input. Also consider how one spiking neuron responses to input frequency variation, which is shown in Fig.4: it can be observed that flat regions exist throughout spiking activity as its unique non-linearity. Therefore, for |ξ| ≤ δ Tinput(fmax−fmin) perturbation does not cause receiving neuron to produce extra spikes. While the exact value of δ changes with different input frequency, it is small only when original input frequency is near the edges of non-linearity. This provides the network with extra robustness to small input perturbations. Based on this, we design the Special Activation Unit (SAU) to be a step function in the form of f (x) = Three baseline networks: MobileNetV2, ResNet101 and DenseNet121 are tested in comparison with SAFE-DNN. We also studied two enhancement methods for baseline networks, namely, training with noisy input (30 dB) and using average filter (2x2) for image pre-processing. Note SAFE-DNN is never trained with noisy images; it is only trained with clean images and only tested with noisy images. Visualization of the embedding space We demonstrate the improved local feature extraction of FD stochastic STDP by comparing capability of the deep network to cluster noisy input. Two SAFEMobileNetV2 are trained with FD stochastic STDP and deterministic STDP, respectively, and tested on noisy input with AWGN noise. The embedding space is taken between the two fully connected layers and each color represents one class. As shown in Fig. 6, 20 dB input is used for (i) and (ii), 18 dB for (iii) and (iv) and 15 dB for (v) and (vi). SAFE-MobileNetV2 implemented with features extracted via FD stochastic STDP provides better clustering of different classes and achieves higher accuracy. Next, we compare the entire SAFE-DNN architecture with alternative designs. First, we consider the standard (baseline) MobileNetV2. The second one, referred to as MobileNetV2-µ, has the same architecture as SAFE-MobileNetV2, but the spiking convolution module is replaced with regular trainable DNN layers. The third one, referred to as the MobileNetV2-λ, is constructed by replacing the activation functions in the first three layers of a trained MobileNetV2-µ with the SAU (without any re-training). The comparisons with MobileNetV2-µ and MobileNetV2-λ show whether benefits of SAFE-MobilenetV2 can be achieved by only architectural modifications or new (SAU) activation function, respectively, without local STDP learning. All networks are trained with CIFAR10 dataset. Fig. 7 shows embedding space visualizations of all four networks with clean and noisy (SNR equal to 25dB) images. We observe that with clean input images, the vectors in embedding space of the baseline MobileNetV2 are distributed into ten distinct clusters. As noise is added to the images the clusters overlap which leads to reduced classification accuracy. On the other hand, SAFE-MobileNetV2 is able to maintained good separation between feature mappings for each class from no noise to 25 dB. We further observe that clusters for noisy images also heavily overlap for MobileNetV2-µ and MobileNetV2-λ, showing that only using architectural modification or spiking activation function, without STDP learning, cannot improve noise robustness of a DNN. Table 2 shows accuracy of all network variants for CIFAR-10. For the baseline DNNs, noise in images significantly degrades classification accuracy. The networks that are trained with noise (30dB noise is used during training) show higher robustness to noise, and the improvement is more prominent when inference noise is at similar level (30 dB) with training noise. For clean images the accuracy is degraded. Average filtering provides accuracy gain over the original networks in highly noisy conditions (less than 20 dB signal to noise ration (SNR)); but major performance drop is observed under mild to no noise. This is expected as average filtering in significant loss of feature details for input images in the CIFAR-10 dataset. For SAFE-DNN implemented with all three DNN architectures, performance in noisy condition is improved over the original network by an appreciable margin. For example, at 20 dB SNR SAFEMobileNetV2 remains at good performance while the original network drops below 40% accuracy, making a significant (50%) gain. Similar trend can be observed for other noise levels. Compared to networks trained with noise, SAFE-DNN shows similar performance at around 30 dB SNR while its advantage increases at higher noise levels. Moreover, for clean images accuracy of SAFE-DNN is on par with the baseline networks. Considering the use case scenario of autonomous vehicles, we conduct test on a subset of ImageNet that contains classes related to traffic (cars, bikes, traffic signs, etc).The subset contains 20 classes with a total of 26,000 training images. The same baseline networks as in the CIFAR10 test are used. Here 25 dB SNR images are used for noise training. The accuracy is shown in Table 3. All networks achieve around 70% top 1 accuracy on clean images. Noise training shows robustness improvement over the baseline network but still negatively affects clean image accuracy. In this test the average filter shows less degradation under no noise condition than for the CIFAR10 test, due to higher resolution of input images. DensNet121 shows more noise robustness than MobileNetV2 and ResNet101 when noise training is used, while for average filtering ResNet101 benefits the most. SAFE-DNN implementations of all three networks exhibit same or better robustness over all noise levels. Clean image classification accuracy is also unaffected. Comparing top 5 accuracy for SAFE-MobileNetV2 and its baselines, as shown in Table 4, SAFE-MobileNetV2 is able to maintain above 80% accuracy even at 5 dB SNR, outperforming all three baselines. Random perturbation We test SAFE-DNN in three more noise structures: Wald, Poisson and salt-and-pepper (SP). For CIFAR10, the is shown in 5. Wald I has a distribution of µ = 3, scale = 0.3 and for Wald II, µ = 13, scale = 1; Poisson I has a distribution with peak of 255 and for Poisson II, 75; S&P I has 5% noisy pixels and S&P II has 20%. Noise-trained DNNs for Wald, Poisson, and SP, are trained with noisy images generated using distributions Wald I, Poisson I, and SP I, resepectively. It can be observed that SAFE-DNN implementation with all three networks are more noise robust than baseline and average filtering. The noise-trained networks performs well when inference noise is aligned with training noise, but performance drops when noise levels are not aligned. Moreover, noise-trained networks trained with mis-aligned noise types performs poorly ( not shown). As for ImageNet subset, networks based on MobileNetV2 are tested. Wald I is a distribution with µ = 5, scale = 0.3 and Wald II is a distribution with µ = 25, scale = 1; Poisson I has a distribution with peak of 255 and for Poisson II, 45; S&P I has 5% noisy pixels and S&P II has 30%. As previously, noise-trained networks are trained with noisy images generated from Wald I, Poisson I and SP I. Similar to previous , as shown in 6 SAFE-MobileNetV2 is more robust to the different noise structures without ever-being trained on any noise structure. Adversarial perturbation We also test SAFE-DNN on adversarial perturbation crafted from black-box adversarial method. For this test, DNNs trained with conventional method are used as target network to generate the perturbed images. The attack method is fast gradient sign method : X adv = X + sign(∇ X J(X, y true)). Here X is the input image, y true the ground truth label and ∈ 0, 255. For source networks that are tested on the perturbed images, DNN trained with different initialization are used as baseline against SAFE-DNN implementation of the deep network. As shown in Table 7, SAFE-DNN also shows improved robustness to noise generated via adversarial perturbations. However, we note that the do not indicate robustness to white-box attacks; and integration of SAFE-DNN with adversarial training approaches will be an interesting future work in this direction. In this paper we present SAFE-DNN as a deep learning architecture that integrates spiking convolutional network with STDP based learning into a conventional DNN for robust low level feature extraction. The experimental show that SAFE-DNN improves robustness to different input perturbations without any prior knowledge of the noise during training/inference. SAFE-DNN is compatible with various DNN designs and incurs negligible computation/memory overhead. Hence, it is an attractive candidate for real-time autonomous systems operating in noisy environment.

represent this text to find a much much longer article about it with say hundreds-thousands of words

A noise robust deep learning architecture.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.


Question: Represent the text to find a very short summary (1-2 sentences max):

Question: Neural embeddings have been used with great success in Natural Language Processing (NLP) where they provide compact representations that encapsulate word similarity and attain state-of-the-art performance in a range of linguistic tasks. The success of neural embeddings has prompted significant amounts of research into applications in domains other than language. One such domain is graph-structured data, where embeddings of vertices can be learned that encapsulate vertex similarity and improve performance on tasks including edge prediction and vertex labelling. For both NLP and graph-based tasks, embeddings in high-dimensional Euclidean spaces have been learned. However, recent work has shown that the appropriate isometric space for embedding complex networks is not the flat Euclidean space, but a negatively curved hyperbolic space. We present a new concept that exploits these recent insights and propose learning neural embeddings of graphs in hyperbolic space. We provide experimental evidence that hyperbolic embeddings significantly outperform Euclidean embeddings on vertex classification tasks for several real-world public datasets. Embeddings are used to represent complex high-dimensional data in lower-dimensional continuous spaces BID28 BID3. Embedded representations provide three principal benefits over sparse schemes: They encapsulate similarity, are compact, and perform better as inputs to machine learning models BID29. These benefits are particularly important for graph-structured data where the native representation is the adjacency matrix, which is typically a sparse matrix of connection weights. Neural embedding models are a flavour of embedding where the embedded representation corresponds to a subset of the connection weights in a neural network (see FIG2), which are learned through backpropagation. Neural embedding models have been shown to improve performance on many tasks across multiple domains, including word analogies (a; BID20, machine translation BID31), document comparison (, missing edge prediction BID12, vertex attribution BID26, product recommendations BID10 BID1, customer value prediction BID14 BID6 and item categorisation BID2 . In all cases, the embeddings are learned without labels (unsupervised) from a sequence of tokens. Previous work on neural embedding models has either either explicitly or implicitly (by using the Euclidean dot product) assumed that the embedding space is Euclidean. However, recent work in the field of complex networks has found that many interesting networks, particularly those with a scale-free structure such as the Internet BID30 BID5 or academic citations BID8 BID7 can be well described with a geometry which is non-Euclidean, such as hyperbolic geometry. Even more recently the problem of mapping graphs and datasets to a low-dimensional hyperbolic space has been addressed in BID24 and BID4. Here we use a neural embedding approach based on the Skipgram architecture to find hyperbolic embeddings. There are two reasons why embedding complex networks in hyperbolic geometry can be expected to perform better than Euclidean geometry. The first is that complex networks exhibit a hierarchical structure. Hyperbolic geometry provides a continuous analogue of tree-like graphs, and even infinite trees have nearly isometric embeddings in hyperbolic space BID11. The second property is that complex networks have power-law degree distributions, ing in high-degree hub vertices. All tiles are of constant area in hyperbolic space, but shrink to zero area at the boundary of the disk in Euclidean space. c Hub and spokes graph. It is impossible to embed this graph in two-dimensional Euclidean space and preserve the properties that all spokes are the same distance from the hub, all spokes are the same distance from each other, and the distance between spokes along the circumference is more than twice the distance to the hub. In hyperbolic space such embeddings exist. FIG1 shows a simple hub-and-spoke graph where each spoke is a distance R from the hub and 2R from each other. For an embedding in two-dimensional Euclidean space it is impossible to reproduce this geometry for more than two spokes. However, in hyperbolic space, large numbers of spokes that satisfy these geometrical constraints can be embedded because the circumference of a circle expands exponentially rather than polynomially with the radius. The starting point for our model is the celebrated Skipgram architecture (a; b) shown in FIG2. Skipgram is a shallow neural network with three layers: An input projection layer that maps from a one-hot-encoded token to a distributed representation, a hidden layer, and an output softmax layer. Skipgram is trained on a sequence of words that is decomposed into (input word, context word)-pairs. The model uses two separate vector representations, one for the input words and another for the context words, with the input representation comprising the learned embedding. The (input word, context word)-pairs are generated by running a fixed length sliding window over a word sequence. Words are initially randomly allocated to vectors within the two vector spaces. Then, for each training word pair, the vector representations of the observed input and context words are pushed towards each other and away from all other words (see FIG2). The model can be extended to network structured data using random walks to create sequences of vertices. Vertices are then treated exactly analogously to words in the NLP formulation. This was originally proposed as DeepWalk BID26. Extensions varying the nature of the random walks have been explored in LINE BID32 and Node2vec BID12.Contribution In this paper, we introduce the new concept of neural embeddings in hyperbolic space. We formulate backpropagation in hyperbolic space and show that using the natural geometry of complex networks improves performance in vertex classification tasks across multiple networks. At the same time, BID24 independently proposed a hyperbolic embedding algorithm that has similarities to ours. The key differences are that BID24 try to fit the hyperbolic distance between nodes using cartesian coordinates in the Poincaré disk, whereas we use a modified cosine distance in a spherical hyperbolic coordinate system. Our approach does not require a numerical constraint to prevent points from'falling off' the edge of the disk and becoming infinitely distant from the others. Hyperbolic geometry emerged through a relaxation of Euclid's fifth geometric postulate (the parallel postulate). In hyperbolic space, there is not just one, but an infinite number of parallel lines that pass through a single point. This is illustrated in FIG1 where every fine line is parallel to the bold, blue line, and all pass through the same point. Hyperbolic space is one of only three types of isotropic space that can be defined entirely by their curvature. The most familiar is flat Euclidean space. Space with uniform positive curvature has an elliptic geometry (e.g. the surface of a sphere) and space with uniform negative curvature has a hyperbolic geometry, which is analogous to a saddle-like surface. Unlike Euclidean space, in hyperbolic space even infinite trees have nearly isometric embeddings, making the space well suited to model complex networks with hierarchical structure. Additionally, the defining features of complex networks, such as power-law degree distributions, strong clustering and community structure, emerge naturally when random graphs are embedded in hyperbolic space.One of the defining characteristics of hyperbolic space is that it is in some sense larger than Euclidean space; the 2D hyperbolic plane cannot be isometrically embedded into Euclidean space of any dimension, unlike elliptic geometry where a 2-sphere can be embedded into 3D Euclidean space etc. The hyperbolic area of a circle or volume of a sphere grows exponentially with its radius, rather than polynomially. This property allows low-dimensional hyperbolic spaces to provide effective representations of data in ways that low-dimensional Euclidean spaces cannot. FIG1 shows a hub-and-spoke graph with four spokes embedded in a two-dimensional Euclidean plane so that each spoke sits on the circumference of a circle surrounding the hub. Each spoke is a distance R from the hub and 2R from every other spoke, but in the embeddings the spokes are a distance of R from the hub, but only R √ 2 from each other. Complex networks often have small numbers of vertices with degrees that are orders of magnitude greater than the median. These vertices approximate hubs. The distance between spokes tends to the distance along the circumference s = 2πR n as the number of spokes n increases, and so the shortest distance between two spokes is via the hub only when n < π. However, for embeddings in hyperbolic space, we get n < sinh R R, such that an infinite number of spokes can satisfy the property that they are the same distance from a hub, and yet the path that connects them via the hub is shorter than along the arc of the circle. As hyperbolic space can not be isometrically embedded in Euclidean space, there are many different representations that each conserve some geometric properties, but distort others. In this paper, we use the Poincaré disk model of hyperbolic space. The Poincaré disk models the infinite two-dimensional hyperbolic plane as a unit disk. For simplicity we work with the two-dimensional disk, but it is easily generalised to the d-dimensional Poincaré ball, where hyperbolic space is represented as a unit d-ball. Hyperbolic distances grow exponentially towards the edge of the disk. The boundary of the disk represents infinitely distant points as the infinite hyperbolic plane is squashed inside the finite disk. This property is illustrated in FIG1 where each tile is of constant area in hyperbolic space, but rapidly shrink to zero area in Euclidean space. Although volumes and distances are warped, the Poincaré disk model is conformal. Straight lines in hyperbolic space intersect the boundary of the disk orthogonally and appear either as diameters of the disk, or arcs of a circle. FIG1 shows a collection of straight hyperbolic lines in the Poincaré disk. Just as in spherical geometry, shortest paths appear curved on a flat map, hyperbolic geodesics also appear curved in the Poicaré disk. This is because it is quicker to move close to the centre of the disk, where distances are shorter, than nearer the edge. In our proposed approach, we will exploit both the conformal property and the circular symmetry of the Poincaré disk. The geometric intuition motivating our approach is that vertices embedded near the middle of the disk can have more'near' neighbours than they could in Euclidean space, whilst vertices nearer the edge of the disk can still be very far from each other. The distance metric in Poincaré disk is a function only of the radius. Exploiting the angular symmetries of the model using polar coordinates considerably simplifies the mathematical description of our approach and the efficiency of our optimiser. Points in the disk are x = (r e, θ), with r e ∈ and θ ∈ [0, 2π). The distance from the origin, r h is given by DISPLAYFORM0 and the circumference of a circle of hyperbolic radius R is C = 2π sinh R. Note that as points approach the edge of the disk, r e = 1, the hyperbolic distance from the origin r h tends to infinity. In Euclidean neural embeddings, the inner product between vector representations of vertices is used to quantify their similarity. However, unlike Euclidean space, hyperbolic space is not a vector space and there is no global inner product. Instead, given points x 1 = (r 1, θ 1) and x 2 = (r 2, θ 2) we define a cosine similarity weighted by the hyperbolic distance from the origin as DISPLAYFORM1 DISPLAYFORM2 It is this function that we will use to quantify the similarity between points in the embedding. We note that using a cosine distance in this way does lose some properties of hyperbolic space such as conformality. Our goal is to learn embeddings that perform well on downstream tasks and the key properties of hyperbolic space that permit this are retained. Trade-offs like this are common in the embeddings literature such as the use of negative sampling BID22 BID20. We adopt the notation of the original Skipgram paper (a) whereby the input vertex is w I and the context / output vertex is w O. The corresponding vector representations are v w I and v w O, which are elements of the two vector spaces shown in FIG2, W and W respectively. Skipgram has a geometric interpretation, shown in FIG2 for vectors in W. Updates to v wj are performed by simply adding (if w j is the observed output vertex) or subtracting (otherwise) an error-weighted portion of the input vector. Similar, though slightly more complicated, update rules apply to the vectors in W. Given this interpretation, it is natural to look for alternative geometries that improve on Euclidean geometry. To embed a graph in hyperbolic space we replace Skipgram's two Euclidean vector spaces (W and W in FIG2) with two Poincaré disks. We learn embeddings by optimising an objective function that predicts context vertices from an input vertex, but we replace the Euclidean dot products used in Skipgram with. A softmax function is used for the conditional predictive distribution DISPLAYFORM0 where v wi is the vector representation of the i th vertex, primed indicates context vectors (see FIG2) and ·, · H is given in. Directly optimising is computationally demanding as the sum in the denominator is over every vertex in the graph. Two commonly used techniques for efficient computation are replacing the softmax with a hierarchical softmax BID21 a) and negative sampling BID22 BID20. We use negative sampling as it is faster. We learn the model using backpropagation with Stochastic Gradient Descent (SGD). Optimisation is conducted in polar native hyperbolic coordinates where r ∈ (0, ∞), θ ∈ (0, 2π]. For optimisation, this coordinate system has two advantages over the cartesian Euclidean system used by BID24. Firstly there is no need to constrain the optimiser s.t. x < 1. This is important as arbitrarily moving points a small Euclidean distance inside the disk equates to an infinite hyperbolic distance. Secondly, polar coordinates in update equations that are simple modifications of the Euclidean updates, which avoids evaluating the metric tensor for each data point. The negative log likelihood using negative sampling is DISPLAYFORM0 where v w I, v w O are the vector representation of the input and context vertices, u j = v wj, v w I H, W neg is a set of samples drawn from the noise distribution and σ is the sigmoid function. The first term represents the observed data and the second term the negative samples. To draw W neg, we specify the noise distribution P n to be the unigram distribution of the vertices in the input sequence raised to 3/4 as in (a). The gradient of the negative log-likelihood in w.r.t. u j is given by DISPLAYFORM1 The derivatives w.r.t. the components of vectors in W (in natural polar hyperbolic coordinates) are DISPLAYFORM2 such that the Jacobian is ∇ r E = DISPLAYFORM3 where χ = w O ∪ W neg, η is the learning rate and j is the prediction error defined in. Calculating the derivatives w.r.t. the input embedding follows the same pattern, and we obtain ∂E ∂r I = j:wj ∈χ DISPLAYFORM4 The corresponding update equations are where t j is an indicator variable s.t. t j = 1 iff w j = w O, and t j = 0 otherwise. Following optimisation, the vectors are mapped back to Euclidean coordinates on the Poincaré disk through θ h → θ e and r h → tanh r h 2. The asymptotic runtimes of the update equations - FORMULA7 and FORMULA0 are the same as Euclidean Skipgram, i.e., the hyperbolic embedding does not add computational burden. In this section, we assess the quality of hyperbolic embeddings and compare them to embeddings in Euclidean spaces. Firstly we perform a qualitative assessment of the embeddings on a synthetic fully connected tree graph and a small social network. It is clear that embeddings in hyperbolic space exhibit a number of features that are superior to Euclidean embeddings. Secondly we run experiments on a number of public benchmark networks, producing both Euclidean and hyperbolic embeddings and contrasting the performance of both on a downstream vertex classification task. We provide a TensorFlow implementation and datasets to replicate our experiments in our github repository 1. To illustrate the usefulness of hyperbolic embeddings we visually compare hyperbolic embeddings with Euclidean plots. In all cases, embeddings were generated using five training epochs on an intermediate dataset of ten-step random walks, one originating at each vertex. FIG3 show hyperbolic embeddings in the 2D Poincaré model of hyperbolic space where the circles of radius 1 is the infinite boundary and Euclidean embeddings in R 2. FIG3 shows embeddings of a complete 4-ary tree with three levels. The vertex numbering is breadth first with one for the root and 2, 3, 4, 5 for the second level etc. The hyperbolic embedding has the root vertex close to the origin of the disk, which is the position with the shortest average path length. The leaves are all located in close proximity to their parents, and there are clearly four clusters representing the tree's branching factor. The Euclidean embedding is incapable of representing the tree structure with adjacent vertices at large distances (such as 1 and 3) and vertices that are maximally separated in the tree appearing close in the embedding (such as 19 and 7). FIG6 shows the 34-vertex karate network, which is split into two factions. FIG6 shows the hyperbolic embedding of this network where the two factions can be clearly separated. In addition, the vertices in FIG6 are the junior instructors, who are forbidden by the instructor (vertex 1) from socialising with other members of the karate club. For this reason they form a community that is only connected through the instructor. This community is clearly visible in FIG6 to the The of our experiments together with the HyBed and 2D Deepwalk embeddings used to derive them are shown in FIG8. The vertex colours of the embedding plots indicate different values of the vertex labels. The legend shown in FIG8 applies to all line graphs. The line graphs show macro F1 scores against the percentage of labelled data used to train a logistic regression classifier with the embeddings as features. Here we follow the method for generating multi-label F1 scores described in BID17. The error bars show one standard error from the mean over ten repetitions. The blue lines show HyBed hyperbolic embeddings, the yellow lines give the 2D Poincaré embeddings of BID24 while the red lines depict Deepwalk embeddings at various dimensions. As we use one-vs-all logistic regression with embedding coordinates as features, good embeddings are those that can linearly separate one class from all other classes. FIG8 shows that HyBed embeddings tend to cluster together similar classes so that they are linearly separable from other classes, unlike the Euclidean embeddings.

Represent the short summary for finding the corresponding scientific article (long).

We learn neural embeddings of graphs in hyperbolic instead of Euclidean space

Represent the TLDR for finding the corresponding scientific article:

This paper proves that skinny neural networks cannot approximate certain functions, no matter how deep they are.


Question: Represent the scientific article to find a short TLDR

Question: The International Competition on Knowledge Engineering for Planning and Scheduling (ICKEPS) plays a pivotal role in fostering the development of new Knowledge Engineering (KE) tools, and in emphasising the importance of principled approaches for all the different KE aspects that are needed for the successful long-term use of planning in real-world applications. In this paper, as an exercise in synthesis and for the sake of stimulating thoughts and discussion, we review the format of previous ICKEPS, to suggest alternative formats for future competitions, ideally to motivate someone to step up and organise the next ones. The International Competition on Knowledge Engineering for Planning and Scheduling (ICKEPS) has been running since 2005 as an almost biennial event promoting the development and importance of the use of knowledge engineering (KE) methods and techniques within this area. The aim of the competition series is to foster developments in the knowledge-based and domain modelling aspects of Automated Planning, to accelerate knowledge engineering research, to encourage the creation and sharing of prototype tools and software platforms that promise more rapid, accessible, and effective ways to construct reliable and efficient Automated Planning systems. The latest competition took place in 2016 1 BID3, which aimed at on-site domain modelling, and highlighted a number of major issues. Most teams did not use any of the existing KE tools, and thus relied only on their expertise. Second, existing tools do not effectively support cooperation, which is needed to cope with the growing complexity of planning applications. Finally, and more worryingly, the number of participants of ICKEPS is still not very large, especially when compared with the latest edition of the International Planning Competition: this suggests that the planning community underestimates the importance of knowledge engineering, despite of its enormous impact on applicability of domain-independent planning in real-world scenarios. Accidental complexity issues BID2, for instance, can prevent the exploitation of automated planning approaches in complex scenarios, and even an unfortunate ordering of elements in the domain model can adversely affect the performance of planning engines BID7.Given the pivotal role played by ICKEPS in promoting the importance of principled KE approaches and tools, we believe it is important to evolve and adapt its format in order to attract and engage a larger number of participants. In this paper, we review the format of past competitions, in order to highlight weaknesses and strengths both from organisers' and participants' perspective. Building on top of this analysis, we suggest some alternative formats that may help future ICKEPS organisers in performing their tasks. It should be noted, though, that the aim of this paper is twofold: to review formats and suggest improvements to ICKEPS, and -more importantly-to make a call for action for organising future competitions focused on KE aspects of planning and scheduling. This section is devoted to describe the formats of past ICK-EPS. The first edition of ICKEPS, held in 2005 BID0, focused on tools for KE. Any tool that helped in knowledge formulation (the acquisition and encoding of domain structure or control heuristics), planner configuration (fusing application knowledge with a Planning or Scheduling engine), validation of the domain model (for example, using visualisation, analysis, reformulation) or validation and maintenance of the Planning and Scheduling system as a whole (for example, using plan/schedule visualisation, or automated knowledge refinement) was allowed to take part. The competition included two stages. In the precompetition stage, the competitors submitted short papers describing the tools. The program committee did light reviewing of the papers with the goal to evaluate relevance of the tools, to send feedback to the competitors, and to contribute to the overall evaluation. During the on-site competition, the participants gave talks about their systems in a workshop-like arrangement, and then they presented the systems during an open demonstration session. Evaluation The tools were evaluated by a jury of experts against the following criteria 2:• The 2007 edition of ICKEPS 3 extended the above format by including an additional simulation stage. A web service including a number of planning and scheduling simulations was made available to participants, in the pre-competition stage, to evaluate their tools. Competitors were made available a short text description of the competition domain, including a description of the simulation API. They used their tools to encode models and submit generated plans for each instance, and received feedback describing the quality of the plan. Evaluation As in ICKEPS 2005, tools were evaluated by judges by taking into account a number of criteria. In 2007, above mentioned criteria were extended by considering also aspects related to the simulation:• domain simulation applicability: how well did the competitors address the simulation domains using their tools? How many domains were the simulators tried on? How long did it take competitors to generate valid plans for the domains? How many problem instances were solved? What was the quality of the plans generated?Specific Tools Design narrowed the scope to tools that support a specific aspect of knowledge engineering technology: those that when input with a model described in an application-area-specific language, output solver-ready domain models. The rationale was to foster the development of tools that can support a rapid exploitation of automated planning in real-world applications, by leveraging on existing planning engines. Evaluation Evaluation was performed by a board of judges that considered a wide range of criteria, divided into two main classes: criteria focusing on the software engineering aspects of the tools (e.g., robustness, usability, etc.), and criteria focusing on the more traditional Planning and Scheduling elements, such as originality, comprehensiveness, etc. The 2012 edition of ICKEPS 4 included two different tracks: The Design Process Track, and the Challenge Track. The former followed the structure of previous ICKEPS, and was focused on the design of both general and specific tools for KE.The newly-introduced challenge track aimed at evaluating the actual usefulness of tools and approaches in tackling complex application domains of Planning and Scheduling. Participants were provided a few months before the actual competition with the natural-language specifications of 3 challenging scenarios for planning and scheduling, and had to tackle one off-site. During the workshop-like demonstration, the participants had to demonstrate the advantage of using their tools/method to produce a model as a solution to the requirements (or a sub-set of) in the specification and the plans for the specified scenarios. The evaluation criteria were the same used in ICKEPS 2009. This format was introduced in ICKEPS 2016, and included two main stages: on-site modelling and a subsequent demonstration. During the first stage, each team received descriptions of 4 scenarios and had to exploit the available time for generating the corresponding models. Scenarios were not taken from real-world applications of planning, but were designed by the organisers taking inspiration from games or from potential application domains. Participants were free to select the scenarios to tackle, and had no restrictions on the number and type of tools that can be used. The only constraints were on the available time -six hours were given-and on the maximum size of teams: at most four members. The day after, each team had to present, in a 10-minute demonstration, the aspects of the knowledge engineering process they exploited for encoding the scenarios. Specifically, teams were expected to discuss: the division of work among team members, the tools used, key decisions taken during the encoding, and the issues they faced. Hard to identify suitable domains and models. On-site modelling More attractive. Can allow to distil good practice in KE.No new tools for the community. Only toy domains can be considered. Table 1: Overview of strengths and weaknesses of considered ICKEPS formats. Evaluation Evaluation included both qualitative and quantitative aspects, and focused on three aspects:• KE tools exploited. This included the list of tools, the KE steps covered by the tools, etc.• Model characteristics. Models were checked in terms of presence bugs, number of operators, readability, etc.• Observed planners' performance. Encoded models were tested using a set of planners, in order to extract useful statistics to be used to empirically compare models. The jury of experts was present at the demonstration, and took into account the above mentioned aspects to award the teams that excelled in all (or in some) of the aspects. In this section we highlight strengths and weaknesses of the format of past ICKEPS, with the aim of synthesising some suggestions for future competitions. An overview of this analysis is provided in Table 1. ICKEPS based on tool design (either specific or general) have fostered the development of a decent number of KE tools, that are exploited by the wider planning community and are extremely helpful for testing planning in real-world applications. The main issue of this format of competition is that, nowadays, quite a significant number of tools is available, and it is hard to provide innovative general tools. Furthermore, the design and development of such tools and techniques in exceedingly demanding, and this has a strong impact on the number of competitors. In the case of general tools, the comparison can also be cumbersome: tools that can support the formulation of models in different languages, or that are aimed at supporting different aspects of KE for planning can be extremely hard to compare. On the other hand, this sort of competitions can be very useful when the focus is on specific aspects of KE / specific languages. This focus can help in the comparison, and can foster the work on some overlooked areas of KE, but may also significantly limit the interest of the community and the number of participants. Off-site modelling and demonstration poses significant burden to the organisers, because they have to identify a set of application domains, or a specific angle within some previously explored domain, where automated planning has not been applied before, and where it is possible to create models that are challenging from a formulation point of view, and at the same time can allow domain-independent planning engines to generate solutions in a reasonable amount of CPU-time. On the other hand, the large amount of time made available to competitors allows to consider more complex cases than those that can be handled in an on-site competition. Furthermore, the more relaxed settings also fosters the use of principled knowledge engineering approaches and techniques. Intuitively, the on-site modelling format has pros and cons which are quite the opposite of what has been discussed for off-site modelling competitions. The limited amount of time available to participants for formulating domain models forces the organisers to consider only "toy" examples, and does not allow to push the boundaries of planning in real-world applications. As observed in ICKEPS 2016, it is usually the case that models are so easy that no KE tool is needed, but a text editor is enough to encode reasonably good models. On the plus side, this style of competition, that is inspired by Hackatons and similar events, lead to a more "funny" sort of competition, and can attract also people that are not usually interested in KE aspects of planning. This is particularly true for students and young researchers. Furthermore, an analysis of techniques exploited by participants can also lead to identify good practice in KE, that can be emphasised and discussed during the demo session or after the competition. Distilling the knowledge obtained by reviewing the formats of past ICKEPS, we found ourselves in the position of suggesting two possible structures for future competitions, that may help to keep alive the interest of the wider ICAPS community in KE aspects, and provide useful data or tools as a tangible heritage. It may be worth to revive formats focusing on what we previously defined as Specific Tool Design. Past ICKEPS exploiting this format considered tools able to translate models between different languages. Taking into account different areas may help to foster the development of tools, and also can help providing some sort of standards that can be used for future work in the area. An example area could be, given also the special attention given to the topic by the 2019 workshop on Knowledge Engineering for Planning and Scheduling (KEPS) 5 (McCluskey et al. 2003), on automated domain model acquisition. With Specific Tool Design focus, it might be possible to specify expected type and amount of input information as well as expected output. Consequently, it might be possible to specify metrics that can be used to quantitatively evaluate a particular tool. Notably, the metrics might follow (soft) constraints that can be set according to a specific application domain, where only some type of data can be expected, or according to some more general "usual" conditions. Such a format can motivate development of tools that might be of a critical importance for advancing the state-of-the-art in the area of KE for Planning and Scheduling. Moreover, with the quantifying metrics the tools can be evaluated more objectively and thus mitigating subjective assessment of judges. Also, to some extent such a format should reduce burden of the organisers as they might focus on narrower scope of the competition. Another suitable format for future ICKEPS can be obtained by mixing off-site and on-site modelling, aiming at exploiting the strengths of both. A suitable combination may be the following: participants are provided with the specifications of "not so easy" application domains to model in a planning language (e.g. PDDL), and demonstrate that existing domain-independent planning engines can handle given sample planning instances from the domains and provide good quality solutions in a reasonable amount of CPU-time. In the on-line stage of the competition, participants will be provided with a modified version of the specifications, and will be required to modify the models accordingly. Again, the domain models will be evaluated on given samples of planning instances. The intuition behind the idea is to foster the exploitation of principled KE approaches for formulating domain models, especially in the off-site stage, with a focus on robustness and maintenance of the models, and possibly help shaping a notion of quality of domain models, such that the model amendment in the on-site stage would be manageable. The potential issue for the organising team is find appropriate and interesting application domains that are both reasonable challenging to formulate, and suitable to "not so easy or hard" modifications. Concluding this paper, we believe that there is a strong need to organise the ICKEPS competitions in order to increase awareness of KE techniques, tool and issues in the ICAPS and general AI communities. The success of future ICK-EPS competitions (e.g. considerable increase of the number of participants) can, in consequence, influence the domainindependent AI planning field by making it accessible for use (by planning non-experts) in various application domains. To give some motivation and inspiration for the future ICKEPS competitions, we, in this paper, provided a review of the format of the past ICKEPS competitions, and suggested two possibly new formats that, we believe, can at-tract more participants and possibly avoid an excessive burden of organisers. We believe that the paper initiates a fruitful discussion about the format of future ICKEPS competitions as well as motivate potential organisers to step up and organise the next competition(s).

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Represent the short summary for finding the corresponding scientific article (long)

In this short paper, we briefly introduce the advantages of using AI planning in Cloud Migration, a preliminary prototype, as well as the chal- lenges the requires attention from the planning and schedul- ing society.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We show that generating English Wikipedia articles can be approached as a multi- document summarization of source documents. We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder- decoder architectures used in sequence transduction. We show that this model can generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual information as reflected in perplexity, ROUGE scores and human evaluations. The sequence-to-sequence framework has demonstrated success in natural-language sequence transduction tasks such as machine translation. More recently, neural techniques have been applied to do single-document, abstractive (paraphrasing) text summarization of news articles BID15, BID9 ). In this prior work, the input to supervised models ranged from the first sentence to the entire text of an article, and they are trained end-to-end to predict reference summaries. Doing this end-to-end requires a significant number of parallel article-summary pairs since language understanding is a pre-requisite to generate fluent summaries. In contrast, we consider the task of multi-document summarization, where the input is a collection of related documents from which a summary is distilled. Prior work has focused on extractive summarization, which select sentences or phrases from the input to form the summaries, rather than generating new text. There has been limited application of abstractive neural methods and one possible reason is the paucity of large, labeled datasets. In this work, we consider English Wikipedia as a supervised machine learning task for multidocument summarization where the input is comprised of a Wikipedia topic (title of article) and a collection of non-Wikipedia reference documents, and the target is the Wikipedia article text. We describe the first attempt to abstractively generate the first section, or lead, of Wikipedia articles conditioned on reference text. In addition to running strong baseline models on the task, we modify the Transformer architecture BID18 to only consist of a decoder, which performs better in the case of longer input sequences compared to recurrent neural network (RNN) and Transformer encoder-decoder models. Finally we show our modeling improvements allow us to generate entire Wikipedia articles. Neural abstractive summarization was pioneered in BID15, where they train headline generation models using the English Gigaword corpus BID3, consisting of news articles from number of publishers. However, the task is more akin to sentence paraphrasing than summarization as only the first sentence of an article is used to predict the headline, another sentence. RNN-based encoder-decoder models with attention (seq2seq) perform very well on this task in both ROUGE BID7, an automatic metric often used in summarization, and human evaluation BID1.In BID9, an abstractive summarization dataset is proposed by modifying a questionanswering dataset of news articles paired with story highlights from Daily Mail and CNN. This task is more difficult than headline-generation because the information used in the highlights may come from many parts of the article and not only the first sentence. One downside of the dataset is that it has an order-of-magnitude fewer parallel examples (310k vs. 3.8M) to learn from. Standard seq2seq models with attention do less well, and a number of techniques are used to augment performance. Another downside is that it is unclear what the guidelines are for creating story highlights and it is obvious that there are significant stylistic differences between the two news publishers. In our work we also train neural abstractive models, but in the multi-document regime with Wikipedia. As can be seen in TAB0, the input and output text are generally much larger, with significant variance depending on the article. The summaries (Wikipedia lead) are multiple sentences and sometimes multiple paragraphs, written in a fairly uniform style as encouraged by the Wikipedia Manual of Style 1. However, the input documents may consist of documents of arbitrary style originating from arbitrary sources. We also show in TAB0 the ROUGE-1 recall scores of the output given the input, which is the proportion of unigrams/words in the output co-occuring in the input. A higher score corresponds to a dataset more amenable to extractive summarization. In particular, if the output is completely embedded somewhere in the input (e.g. a wiki-clone), the score would be 100. Given a score of only 59.2 compared to 76.1 and 78.7 for other summarization datasets shows that ours is the least amenable to purely extractive methods. There is a rich body of work incorporating Wikipedia for machine learning tasks, including questionanswering BID4, BID13 ) and information extraction BID6, and text generation from structured data BID5.The closest work to ours involving generating Wikipedia is BID16, where articles are generated extractively (instead of abstractively in our case) from reference documents using learned templates. The Wikipedia articles are restricted to two categories, whereas we use all article types. The reference documents are obtained from a search engine, with the Wikipedia topic used as query similar to our search engine references. However we also show with documents only found in the References section of the Wikipedia articles. Previous work on neural abstractive summarization relies on RNNs as fundamental modules, mirroring techniques successful in machine translation (MT). Recently, state-of-the-art MT were obtained using a non-recurrent architecture, called the Transformer BID18. The lack of recurrence enables greater within-training-example parallelization, at the cost of quadratic complexity in the input sequence length. We find the Transformer transfers well to medium length, input sequence summarization and describe modifications to better handle longer sequences. from the Google search engine, using the article section titles as queries. For each query, we collect 10 pages. From this collection we remove the Wikipedia article itself, which is often among the top . We also remove "clones", which are detected when there is a high-level of unigram overlap with the article (details provided in A.2.1). We denote these refined search for an article, a i, as S i ⊂ D. Similar to C i, we extract only the text to use as input. TAB1 describes overall properties of our WikiSum dataset. Many articles have few citations, motivating our supplementation of the source documents with web search . On the other hand, citations when available, tend to be of higher-quality. When counting the total words in the entire dataset, it is orders-of-magnitude larger than previous summarization datasets. To have consistent train/development/test data across corpus-comparison experiments, we restrict the articles to those with at least one crawlable citation. We divide the articles roughly into 80/10/10 for train/development/test subsets, ing in 1865750, 233252, and 232998 examples respectively. Because the amount of text in input reference documents (C i, S i) can be very large (see TAB1) it is infeasible to train an end-to-end abstractive model given the memory constraints of current hardware. Hence, we first coarsely select a subset of the input using extractive summarization. The second stage involves training an abstractive model that generates the Wikipedia text while conditioning on this extraction. This two-stage process is inspired by by how humans might summarize multiple long documents: First highlight pertinent information, then conditionally generate the summary based on the highlights. We investigate three extractive methods from the summarization literature, along with a trivial and cheating method, to assess the importance of this stage. For each article, a i we create a ranked list of paragraphs, {p DISPLAYFORM0 is the rank of the jth paragraph p i j of (C i, S i). From this we select the first L tokens as input to the second abstractive stage.1. Identity: As a trivial baseline extractor, we simply use the first L tokens of the input.2. tf-idf: A non-trivial ranking is to consider ranking paragraphs as documents in a queryretrieval problem, where the query is the title of the article, T (a i). We compute tf-idf BID14 for the query, with respect to the documents, {p i j}. That is, we summate for each word in the query DISPLAYFORM1 where N w, N d, and N dw are the count of the word in the document, total number of documents, and total number of documents containing the word, respectively.3. TextRank BID8: A weighted graph is defined where text units are nodes and edges are defined by a similarity measure based on word overlap. An algorithm similar to PageRank BID11 is then used to compute the ranking of text units. We used paragraphs for the text units.4. SumBasic BID10: Word frequencies in the input text are used to assign scores to words, which are in turn used to score sentences. After selecting the best scoring sentence, words in it have their scores reduced, and the process is repeated until the desired summary length is reached. To further demonstrate the quality of extraction on the final performance, we implement a cheating extractor that ranks {p i j} using recall of bigrams in the ground truth text: DISPLAYFORM0 4.2 ABSTRACTIVE STAGE 4.2.1 DATA REPRESENTATION Given the ordered paragraphs {p i Ri(j) }, we derive the raw text input simply as the concatenation of the paragraphs in order, the most relevant at the beginning, and prefixed with the title. We then encode the text using sub-word tokenization similar to BID19 with a vocabulary size of 32,000 yielding tokenized input, x i: DISPLAYFORM1 For various values of L in experiments, we truncate the tokens to form the input sequence: DISPLAYFORM2 For the output, we use the same vocabulary and tokenization for the Wikipedia lead text but do not do any truncation across experiments. Next we describe the abstractive models, W, that learn to write articles, a i = W (m L i), which we treat as a sequence transduction problem from very long input sequences (up to L = 11000) to medium output sequences (typically less than 500). As a baseline we apply the standard LSTM encoder-decoder with attention (seq2seq-att) as in BID0 to this task. As is typical we train to optimize the maximum-likelihood objective: DISPLAYFORM0 A stronger, more recent baseline that we use is the non-recurrent Transformer model described in 2.3, which also has symmetric encoder and decoder modules (T-ED). We introduce a simple but effective modification to T-ED for long sequences that drops the encoder module (almost reducing model parameters by half for a given hyper-parameter set), combines the input and output sequences into a single "sentence" and is trained as a standard language model., where δ is a special separator token and train a model to predict the next word given the previous ones: DISPLAYFORM0 Since the model is forced to predict the next token in the input, m, as well as y, error signals are propagated from both input and output time-steps during training. We also suspect that for monolingual text-to-text tasks redundant information is re-learned about language in the encoder and decoder. We believe this allows for easier optimization and empirically observe this with longer sequences (see Section 5.3). Note that because of the self-attention of the Transformer, when generating the next token, attention from both m and y are considered. At inference we provide the input sequence, m i, initially, and auto-regressively generate the output, y i, as normal. To re-use the terminology used to describe the Transformer, the attention is a function of a query (Q) and set of key (K) and value (V) pairs. To handle longer sequences, we modify the multi-head self-attention of the Transformer to reduce memory usage by limiting the dot products between Q and K in: DISPLAYFORM0 Local attention: Sequence tokens are divided into blocks of similar length and attention is performed in each block independently. As the attention memory cost per block becomes constant, this modification allow us to keep the number of activations linear with respect to the sequence length. In our experiments, we choose to have blocks of 256 tokens. Memory-compressed attention: After projecting the tokens into the query, key, and value embeddings, we reduce the number of keys and values by using a strided convolution. The number of queries remains unchanged. This modification allows us to divide the number of activations by a compression factor. In our experiments we use convolution kernels of size 3 with stride 3. In contrast to local attention layers, which only capture the local information within a block, the memorycompressed attention layers are able to exchange information globally on the entire sequence. These modifications (see FIG0) allow us in practice to process sequences 3x in length over the T-D model. For both local and memory-compressed attention, masking is added to prevent the queries from attending to future keys and values. Our final architecture is a 5-layer network (LMLML) alternating between local-attention (L) layers and memory-compressed attention (M) layers (in BID18 it is 6 identical layers). We also added in some experiments one mixture of experts (MoE) layer to increase the network's capacity. In experiments we evaluate based on perplexity (per-wordpiece), a common language modeling metric, and ROUGE-L F1 (version ROUGE-1.5.5), a common metric used in comparing candidate and reference summaries. Note the F1 flavor of ROUGE is more appropriate in this setting as we do not explicitly constrain the output length in abstractive models; it is the harmonic mean of ROUGERecall (which favors long summaries) and ROUGE-Precision (which favors short summaries).Although optimizing ROUGE directly has been shown to not always yield the best summaries as evaluated by human judgment BID12, we found that for our task optimizing for perplexity correlates with increased ROUGE and human judgment. We suspect that the relatively uniform style of Wikipedia articles makes ROUGE more appropriate here than in general abstractive summarization tasks. For all abstractive model training, we use the open-source tensor2tensor 2 library. The seq2seq baseline had a hidden size of 128 with 2 layers (we use the hyper-parameter set defined in the library as lstm attention).For the Transformer encoder-decoder (T-ED), we use the hyper-parameter set transfomer base v1 and train for 1 million steps. Models exhibited very little overfitting and did not require early-stopping. The Transformer Decoder (T-D) was identical to the decoder part of T-ED. The T-DMCA model is similar to T-D, but with the enhancements described in section 4.2.4.Unless otherwise stated, during decoding we use a beam search of size 4 and length penalty α = 0.6 BID19 and decode until an end-of-sequence token is reached. Extractive-only is not enough: We investigate performance of extractive methods without the abstractive model by looking at the ROUGE-L F1 scores after running tf-idf, SumBasic, and TextRank in Figure 2, without any abstractive model. In the case of TextRank and SumBasic we matched the output length to the target length and observe the extractive methods perform roughly in-line with each other in terms of ROUGE-L F1. Our best abstractive model more than doubled this metric. Further, this model yields large improvements in perceived linguistic quality (elaborated below).Extractive method: From TAB2 we observe that smart extraction is critical for final abstractive performance. There is a significant gap between doing nothing, identity, and extractive summarization, tf-idf. Further, there is a significant gap between tf-idf and the cheating extractor, suggesting future work in improving the extraction step could in significant improvements. One possibility is to train a supervised model to predict relevance (Eq. 1), which we leave as future work. For subsequent experiments we fix the extractive method to tf-idf. Input Corpus: From table 3 we also observe that, unsurprisingly, the combined dataset performs best, but the gaps between it and using only one of citations or search are both significant and their contributions are complementary. In subsequent experiments, we report only the combined . Abstractive model architecture and input length: As we see from TAB3, seq2seq-attention as a baseline does quite poorly on this task compared to the Transformer architectures. As seen in FIG2, we observe that the Transformer encoder-decoder, T-ED, architecture consistently improves in performance until a best of around L = 500 − 1000 and is unable to learn at L = 2000. This motivated the Transformer-Decoder, which we found could learn and improve up to L = 4000, before running out of memory on our machines equipped with 16GB of GPU RAM (NVIDIA P100). By using the T-DMCA modifications, we were able to train up to L = 11000 and continued to see improvements in performance. We also found the MoE-layer helped performance by adding model capacity at high L, for example dropping log-perplexity from 2.05 to 1.93 at L = 11000 with 128 experts. Our best model attempted uses 256 experts at L = 7500 (we were unable to use 256 experts with L = 11000 due to memory constraints) and achieves a perplexity of 1.90, Human Evaluation -Linguistic quality We conducted a DUC-style human evaluation of linguistic quality 3 of samples from a baseline abstractive (seq2seq), the best extractive (tf-idf), and our best T-DMCA models. Five different dimensions are assessed: grammaticality, non-redundancy, referential clarity, focus, and structure/coherence. As seen in Table 5, the T-DMCA model does statistically significantly better on all dimensions, except on non-redundancy where tf-idf does about as well. Overall, we observed high fluency and coherence from our best abstractive model. Occasionally we observed some repetition of phrases which hurt the non-redundancy and structure, but it was much rarer compared with the other abstractive method, seq2seq. The biggest weakness of the extractive Table 5: Linguistic quality human evaluation scores (scale 1-5, higher is better). A score significantly different (according to the Welch Two Sample t-test, with p = 0.001) than the T-DMCA model is denoted by *. Focus Grammar Referential clarity T-DMCA (best) 4.5 4.6 4.2 4.5 4.2 tf-idf -only 3.0* 3.6* 3.9 3.2* 2.7* seq2seq-attention 3.0* 3.4* 2.1* 3.4* 2.3* Table 6: Side-by-side for two models pair with large automatic metric gaps DISPLAYFORM0 38.8 1.5 method compared with our best abstractive model was the lack of structure and coherence in the summaries. Human Evaluation -side-by-side preference We validated our chosen metrics correlate with human preference by conducting two side-by-side human evaluation experiments, comparing models with large gaps in perplexity/ROUGE. We observe in Table 6 that human judgment correlates with our automatic metrics, but it becomes more difficult to distinguish at the higher-end of model performance. Details of the human evaluation experimental designs can be found in Appendix A.3.To summarize the quantitative , we believe the highest impact future work will be from improving the extractive stage and extending the decoder-only architectures to learn from larger L while maintaining sufficient model capacity. Comparison with BID16: A direct comparison with BID16 is difficult for three reasons: (a) they report only for two small subsets of Wikipedia, Diseases and American Actors; (b) we report on lead generation instead of full-articles; (c) we were unable to obtain the exact articles they used as input and output (in particular they make no claim of Wikiclone detection). However, we make a best-effort comparison by finding the subset of articles of our test set that correspond to Diseases and American Actors, the two categories reported on by Sauper & Barzilay and reporting our ROUGE-1 scores TAB4. We observe that we perform better on American Actors than Diseases, probably because of the prevalence of the former (and biographies) in Wikipedia compared to the latter in our training set for our single, global model, whereas Sauper & Barzilay likely benefit from the category-specific templates. On average our ROUGE-1 scores are higher but do worse on the less common and somewhat specific disease category. In FIG3, we show the predictions from three different models (using tf-idf extraction, and the combined corpus) along with the Wikipedia ground truth. As the perplexity decreases we see improvements in the model outputs, in terms of fluency, factual accuracy, and narrative complexity. In particular, the T-DMCA model offers a respectable alternative to the Wikipedia version and is more succinct, while mentioning key facts, such as where the law firm was located, when and how it was formed, and the rise and fall of the firm. In manual inspection of model outputs, we noticed an unexpected side-effect: models learn to translate names from English into multiple languages, e.g. Rohit Viswanath into Hindi (see FIG4). Although we did not do a systematic evaluation of the translations, we found they are often correct, and often they are not found in the Wikipedia article itself. We also verified that in general the translation is not merely copied from the source, such as example cases where the target language is the incorrect one (e.g. translation of an English name into Ukrainian). Given that we have shown it is possible to learn sequence transduction models on combined inputoutput sequence lengths of approximately 12000 using the T-D architecture, we show that it is possible to train a model to generate entire Wikipedia articles. As a preliminary , we trained two T-DMCA models: One is trained to use L = 6000 reference tokens to predict at most 2192 article tokens (longer examples are ignored) and another is conditioned only on the title and generates articles up to 4000 tokens long. We show samples from both models in Appendix A.1. Although the generated articles are not as good as the real Wikipedia or our lead section samples, the models can be seen to organize the article into plausible sections and exhibit global coherence over multi-paragraph text. The model with access to reference documents inserts factual information in the generated article. Although we did not focus or tune on the full-article task we see this as an interesting future work for abstractive summarization. We have shown that generating Wikipedia can be approached as a multi-document summarization problem with a large, parallel dataset, and demonstrated a two-stage extractive-abstractive framework for carrying it out. The coarse extraction method used in the first stage appears to have a significant effect on final performance, suggesting further research on improving it would be fruitful. We introduce a new, decoder-only sequence transduction model for the abstractive stage, capable of handling very long input-output examples. This model significantly outperforms traditional encoderdecoder architectures on long sequences, allowing us to condition on many reference documents and to generate coherent and informative Wikipedia articles. To encourage further research on large-scale summarization, we will release the URLs used in our experiments (the Wikipedia URL as well as the URLs of its references). We also provide code that that extracts content from the CommonCrawl dataset 4, which is freely available for download. We use the open-source tensor2tensor 5 library for training abstractive models and will be releasing our abstractive modeling code extensions. Further details are available at https:// goo.gl/wSuuS9. To assess linguistic quality, we randomly selected samples generated by models from the test set and ask raters to choose a score from 1 to 5 (higher is better) for five dimensions: Grammaticality, Non-redundancy, Referential clarity, Focus, and Structure and Coherence. These were used in the past at DUC for evaluating summaries BID2. For each model we selected 25 examples and averaged the scores for each question across 3 raters (out of pool of 7).To compare two models by human evaluation, we randomly select examples from the test set and show model outputs side-by-side in the interface shown in Figure 9. Which side a model appears on is randomized per example and rater. For the experiments in Table 6 we had 3 raters score 25 examples each and computed the ratio of ratings preferring one model over the other. A.4 EXAMPLE ABSTRACTIVE MODEL INPUT Figure 9: Screenshot of side-by-side human evaluation tool. Raters are asked whether they prefer model output on the left or right, given a ground truth Wikipedia text. Figure 10: Example extractive-output/abstractive-input for models in "dewey & lebeouf" example. The extractive method used is tf-idf.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We generate Wikipedia articles abstractively conditioned on source document text.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.