Question: Represent the scientific article to find a short TLDR.

Question: This paper puts forward a new text to tensor representation that relies on information compression techniques to assign shorter codes to the most frequently used characters. This representation is language-independent with no need of pretraining and produces an encoding with no information loss. It provides an adequate description of the morphology of text, as it is able to represent prefixes, declensions, and inflections with similar vectors and are able to represent even unseen words on the training dataset. Similarly, as it is compact yet sparse, is ideal for speed up training times using tensor processing libraries. As part of this paper, we show that this technique is especially effective when coupled with convolutional neural networks (CNNs) for text classification at character-level. We apply two variants of CNN coupled with it. Experimental show that it drastically reduces the number of parameters to be optimized, ing in competitive classification accuracy values in only a fraction of the time spent by one-hot encoding representations, thus enabling training in commodity hardware. Document classification is one of the principal tasks addressed in the context of natural language processing BID22. It implies associating a document -or any text fragment, for that matter-with a category or label relying on their content. The increasing availability of texts in digital form, especially through the Internet, has called for the development of statistical and artificial intelligence tools for automating this process. Spam detectors, sentiment analysis, news archiving, among many others, demand high-quality text classifiers. There is a broad range of approaches to document classification (see BID22 BID1 BID11 BID15). An important portion of them relies on a representation that handles words as the atomic element of text. Consequently, those methods carry out their analysis through statistics of words occurrence. However, the variability of words and structures belonging to a language hinders the viability of this method. That is why, these models have a superior performance in specific domains and applications, where the vocabulary is or can be restricted to a relatively small number of words, possibly chosen by a specialist. Furthermore, such modeling becomes specific to a language, causing the replication process in another language to be carried out from scratch.In recent years, we have experienced a revolution in the machine learning with the advent of deep learning methods BID8. The development of convolutional neural networks (CNNs) BID17 coupled with the popularization of parallel computing libraries (e. g. Theano BID2, Tensorflow BID0, Keras BID7, etc.) that simplify general-purpose computing on graphics processing units (GPGPU) BID21 has been successful in tackling image classification problem BID16 quickly becoming the state of the art of the field. As it could be expected, the success of deep learning and CNNs in the image classification domain has prompted the interest to extend the deep learning principles to the document classification domain. Some existing methods have been updated but the clear majority are still based on the to-kenization of words and the inference of their statistics. Bag of Words (BoW) BID13 and Word2vec BID20 are some of the most popular strategies. It can be argued that the replication of image classification success in the documents domain faces as main challenge the difficulty of representing text as numerical tensors. To address this issue, suggested a groundbreaking approach that considers the characters as the atomic elements of a text. In particular, they represented the text as a sequence of one-hot encoded characters. This encoding provides a robust, language-independent representation of texts as matrices, that are then used as inputs of different CNNs. Their experimental showed that this approach was able to attain and, in some cases, improve the state of the art in complex text classification problems. More recently, BID25 improved those by combining CNNs with Long Short-Term Memories (LSTMs) BID10. In spite of that, the impact of this idea is hampered by the large computational demands of the approach, since its training can take days per epoch in relatively complex problems. Character-level representations have the potential of being more robust than word-level ones. On the other hand, they are computationally more expensive because detecting syntactic and semantic relationships at the character-level is more expensive . One possible solution could be a word representation that incorporates the character-level information. In this paper, we propose an efficient character-level encoding of word to represent texts derived from the Tagged Huffman BID23 information compression technique. This encoding takes into account the character appearance frequency in the texts in order to assign shorter codes to the most frequently used ones. This novel text encoding makes the idea put forward by more computationally accessible by reducing its training requirements in terms of time and memory. The proposed encoding makes possible to represent larger portions of texts in a less sparse form, without any loss of information, while preserving the ability to encode any word, even those not present in the training dataset ones. In order to study the impact of this encoding, we coupled it with two CNN architectures. The experimental studies performed showed that we managed to achieve a performance similar or in some cases better than the state of the art at a fraction of the training time even if we employed a simpler hardware setup. Our main contribution is to show that this novel character-level text encoding produces a reduced input matrix, leading to a substantial reduction in training times while producing comparable or better in terms of accuracy than the original approach by. This opens the door to more complex applications, the use of devices with lower computational power and the exploration of other approaches that can be coupled with input representation. The rest of the paper is structured as follows. In the next section, we deal with the theoretical foundations and motivation that are required for the ensuing discussions. There we also analyze the alternatives to character-level text compression that were taken into account for producing our proposal. After that, in Section 3, we describe the encoding procedure and the neural network architectures that will take part of the experiments. Subsequently, in Section 4, we replicate the experiments of in order to contrast our proposal with theirs under comparable conditions. Finally, in Section 5, we provide some final remarks, conclusive comments and outline our future work directions. The success in using Convolutional Neural Networks (CNNs) BID17 in image classification BID16 flourish with the development of many libraries BID6 BID7 BID2, techniques and hardware. The effort to use CNNs for text classification tasks is justified by the possibility of appropriating these tools for obtaining better and more robust algorithms, facilitating the use of the same approach for several applications. There are two usual approaches to use CNNs for handling textual information: the (i) bag of words (BoW) BID13 and (ii) Word2vec BID20 approaches. In the case of BoW and some of its variants, for representing each word in a vocabulary of size N, a digit 1 is placed in the correspondent position of that word in a 1 × N vector, all others positions remaining with digit 0. Since natural languages usually have a large vocabulary, a limited subset of the vocabulary must be used in order to make viable to perform the necessary computations in terms of memory requirements. The chosen subset of the vocabulary must be representative of the texts. Therefore, in practical problems, a great deal of attention is devoted to this matter. In particular, it is common to involve an application domain specialist or use some kind of word frequency statistic or relevance metric, where the most frequent and rare words are excluded. In the word2vec approach, each word is projected via a metric embedding of fixed size, representing its co-occurrence in a large corpus of text in the same language of the texts of interest. It is possible to use pretrained vectors or readjust the representation with new words. The main problem with both strategies is that it does not allows to represent words that are not in the training dataset. Typos, spelling errors, mixed up letters and text written in languages with a complex structure (declensions, conjugations, etc.) are completely ignored. Establishing the character as the basic unit of text formation provides a better opportunity to be robust to typos, acceptance of neologisms, and other textual forms as equations and chemical formulas, abbreviations and idiosyncrasies of written language on the internet, such as emoticons, slang and dialects of the online world, etc. Assuming the word as a base item, much of this ability is lost, especially when models assume that text producers use a formal language. put forward an important innovation in this regard. They represent text as a sequence of characters, not words. Consequently, they are capable of reducing the vocabulary of symbols to the size of the alphabet (69 in the case of the paper) and thus allowing the use of one-hot encoding BID9. In this paper, they represented a text as a matrix of size 1014 × 69 where each row corresponds to a position in the text sequence and columns with the presence or not of a given character. Therefore, a row with a 1 in a given column indicates that the presence of the corresponding character in that point of the text. With this representation on hand, they applied a CNN and obtained competitive with other techniques, and in some cases improving the state of the art. However, the main drawback is a large computational requirement, that in some cases called for days per training epoch. The obtained by them suggest that language, and therefore, text, can be treated as a sequence of signals, like any other. However, the training times and the dimension of the matrices to be computed are still obstacles to the most effective use of the method. That is why a better encoding of text could be a right path towards a substantial improvement if this issue. Searching for a way to reduce these training time while retaining the flexibility and power of character-level convolutional to classify text, we found a way to better encode texts. Our approach, with competitive accuracy, achieve a significant reduction in time execution from hours to minutes and from days to hours, by epoch. To achieve this performance in execution, our approach consisted of two elements:• Obtaining a shorter representation: At first, we though of using some form of encoding each character, and use the same approach of but we realize that using a variable length encoding for each word could be more efficient. To do this we need a way to encode each char, generating distinct concatenated code for representing each word and that words with the same prefix were near each other, especially to respond well to declensions.• Obtaining a sparse representation: To take full advantage of libraries of tensor multiplications like NVidia CuDNN BID6, we need a representation that was sparse, so our code should be composed of many 0s and a few 1s. Although the Huffman encoding BID12 yields shortest possible, it does not generate unique representations once we concatenate encoded characters to form a word. We investigated encoding of BID5 and found promising alternatives. Our approach is based on the Tagged Huffman encoding (Silva de BID23, where a pair of '0' digits is the signal and the digit '1' is the tag of beginning and end code, the only difference is that for our approach, we need a shorter version to reduce the size of input matrix, so we choose to use only one digit 0 instead of two for each char, marking the beginning and the end of each char code with a digit 1, the same way. As in the approach by Silva de , the coding we employ has the following advantages BID5 : DISPLAYFORM0 No character code is a prefix of another, that is, the match is one-to-one.2. It allows a direct search, that is, to search for a word in an encoded document, just encode the word and use traditional methods of comparing strings with the encoded document.3. This encoding approach is a compression technique, so it also allows saving already encoded text documents permanently using a binary system, requiring less storage space. These advantages become attractive especially if the goal is to extract knowledge about files in a repository to perform various classifications on different dimensions with the same files. A possible advantage of this encoding over others strategies that use a word as an atomic representation of text is better respond to unseen words on training data, once that the network at least have some prefix to guess the meaning of the word, the same way we humans do. This is especially interesting in languages where there are a lot of declensions like Portuguese, Spanish, Italian, French, Russian, German, Arabic and Turkish, for example. The coding procedure is not restricted to any size of vocabulary, the only problem is that less frequent characters will generate bigger codes, consequently, bigger encoding matrix. If your database has a lot of them, you could use a higher word code size. Our main contribution is demonstrate that such approach reduces the dimensionality of the encoded matrix, substantially reducing training time and allow the use of devices with lower computational power, remaining with competitive accuracy. In all of our experiments, we used the following procedure to encode words:• Obtaining a character frequency rank: We read the text database and count the frequency of each character, generating a list sorted by frequency of occurrence. Then we create a rank with only the relative positions of the characters. For a given language, this rank is quite stable since only the order of the rank is used. This means that if all documents are in the same idiom, this procedure can be replaced by a list with the characters rank of frequency for that language.• Creating a mapping from character to compress code: To encode each character, we insert the digit 0 in the same amount of the rank position of the character, between a 1 digit to signal the begin and another 1 digit to signal the end of the code. TAB0 have some examples of encoded characters. To encode each word, we just concatenate the codes of each character. As an example, we provide in TAB1 some examples of plain text words and their corresponding encoding. Given a document, we consider that it is composed of words, being those any set of characters limited by the space character. This means 'word' could be math equations, web addresses, L A T E X code, computer programming commands, etc. In TAB1 we can see that this encoding could represent words with the same prefix in vectors that have same initial coordinates. Slangs like tl;dr : too long, did not read and u2 : you too as well mathematical expressions like e a could be manipulated. In a matrix of number of words × code size representing the document, each row represents a properly encoded word, where the code is embedded with its first symbol in the first column. Columns that were not occupied are padded with 0, larger codes are represented up to the chosen limit. Unoccupied lines are filled with 0 and larger documents are represented only up to the chosen maximum number of words, ignoring the last remaining ones. As an example, we represent a document in an 8 × 65 matrix in Figure 1 . DISPLAYFORM0 In the example in Figure 1 we used a 8 × 65 matrix (520 elements) to encode the text with a certain amount of slack. At the very least, we would need 7 × 64 (448 elements). In contrast, the approach of would employ at least 32×69 elements to represent the same sentence. In our experiments, 256 coordinates were enough to represent 99.5% of the words from one of the databases studied. In all datasets studied in this paper, we choose 128 as a limit of words to represent a document, encoding each text in a 128 × 256 matrix. As mentioned before, this work was prompted by the of. In their original approach, they encode each character using one-hot encoding in a vocabulary of 69 elements. The non-space characters are letters, numbers and punctuation. The model is composed of 9 layers, 6 of convolutions and 3 fully connected. Their architecture is described in TAB2.They used stochastic gradient descent (SGD) with a minibatch of size 128, using momentum 0.9 and initial step size 0.01 which is halved every 3 epochs for 10 times. So, their were obtained in at least 30 epochs. To verify the efficiency of the encoding procedure, we realized 3 experiments, named CNN1, CNN2 and LSTM: At first, we choose a network architecture that we usually use to classify text using an embedding created by word2vec BID20, the only difference is that instead of 300 features, we reduce the input size to 256. This architecture we named CNN1. It is based on concatenation of convolutions in a shallow way, inspired by work of BID14, who achieve state of the art for some databases. We trained this model for 5 epochs. The neural network architecture CNN1 is summarized in Figure 2a as a diagram. Prompted by the positive outcome of CNN1, we decided to investigate others possible architectures. We created another shallow but wide convolution architecture following the recommendations of BID29 for choosing parameters, executing training on dataset ag news, for being the smallest of our test datasets. This architecture is composed by:• Convolution width filter: a combination of region sizes near the optimal single best region size outperforms using multiple region sizes far from the optimal single region size BID29. We scan the width from 1 to 7 comparing accuracy performance. In these evaluations, the convolution of width 1 was a better option.• Pooling size: max pooling consistently performs better than alternative strategies for the task of sentence classification BID29.The neural network architecture CNN2 is summarized in Figure 2b as a diagram To illustrate the possibilities of applying the proposed encoding, we did a experiment using LSTMs BID10, similar to LSTM model described by, the difference is that instead of using word2vec embedding BID20, we used our representation. The architecture is very simple: an input layer of 126 × 256, followed by a LSTM layer of 300, a Dropout layer of.10 BID24, a fully connected layer of 128 units and a softmax layer. We trained this model for 5 epochs. This architecture is in general, twice slower than CNN1. The neural network architecture LSTM is summarized in Figure 2c as a diagram. An essential part of this work is to contrast our proposal both within the context of character-level text classification and with other state-of-the-art approaches. The databases used were the same as those cited in an article by, where there is an extensive description of them. 1 A detailed analysis of these datasets is out of the scope of this paper, instead, we will only summarize the main characteristics:• AG's news: categorized news articles from more than 2000 news sources. Four classes (World, Sports, Business, SciTech).The dataset contains 120k train samples and 7.6k test samples equally distributed.• Sogou news: categorized news articles originally in Chinese. applied the pypinyin package combined with jieba Chinese segmentation system to produce Pinyin -a phonetic romanization of Chinese. Five classes (sports, finance, entertainment, automobile and technology). The dataset contains 450k train samples and 60k test samples equally distributed.• DBpedia: title and abstract from Wikipedia articles available in DBpedia crowd-sourced community BID18. Fourteen non-overlapping classes from DBpedia 2014. The dataset contains 560k train samples and 70k test samples equally distributed.• Yelp full: sentiment analysis from the Yelp Dataset Challenge in 2015 2. Five classes representing the number of stars a user has given. The dataset contains 560k train samples and 38k test samples equally distributed..• Yelp polarity: sentiment analysis from the Yelp Dataset Challenge in 2015 2. The original data is transformed into a polarity problem. Rating of 1 and 2 stars are represented as Bad and 4 and 5 as Good. The dataset contains 560k train samples and 50k test samples equally distributed... • Amazon full: sentiment analysis from Amazon reviews dataset from the Stanford Network Analysis Project (SNAP) BID19. Five classes representing the number of stars a user has given. The dataset contains 3,000k train samples and 650k test samples..• Amazon polarity: sentiment analysis from Amazon reviews dataset from the Stanford Network Analysis Project (SNAP) BID19. Two classes, rating of 1 and 2 stars are represented as Bad and 4 and 5 as Good. The dataset contains 3,600k train samples and 400k test samples equally distributed..The baseline comparison models are the same of where there is an extensive description of them, we just reproduce their , the only difference is that they report loss error and for better comprehension, we translated it to accuracy. In there is an extensive description of them. In this paper, we just summarize the main information:• Bag of Words (BoW) and its term-frequency inverse-document-frequency (BoW TFIDF): For each dataset, they selected 50,000 most frequent words from the training subset. For the normal bag-of-words, they used the counts of each word as the features and for the TFIDF they used the counts as the term-frequency. • Bag-of-ngrams (Ngrams) and its TFIDF (Ngrams TFIDF): The bag-of-ngrams models were constructed by selecting the 500,000 most frequent n-grams (up to 5-grams) from the training subset for each dataset. The feature values were computed the same way as in the bag-of-words model.• Bag-of-means on word embedding: an experimental model that uses k-means on word2vec BID20 learned from the training subset of each dataset, and then used these learned means as representatives of the clustered words. Took into consideration all the words that appeared more than 5 times in the training subset. The dimension of the embedding is 300. The bag-of-means features are computed the same way as in the bag-of-words model. The number of means is 5000.• Long Short Term Memory (LSTM): The LSTM BID10 model used is word-based, using pretrained word2vec BID20 embedding of size 300. The model is formed by taking a mean of the outputs of all LSTM cells to form a feature vector, and then using multinominal logistic regression on this feature vector. The output dimension is 512.For all the experiments, we used the environment and parameters settings listed in TAB3. Besides the encoding procedure, we do not use any preprocessing strategy except the use of lowercase letters. No data enhancement technique was employed. All the and comparison with traditional models and the approaches of is shown in TAB4.As tabular data are a hard to grasp, we have decided to go for a graphical representation of the . In particular, from the previous , we have selected the large and small architectures of and our CNN1, CNN2 and LSTM. Those values of accuracy were then scaled to the interval for every dataset, being 0 the worst and 1 the best performance among all models, including tradicional models. The outcome of this process is represented on Figure 3. On TAB5, we did a running time comparison, based on reports available by.The main objective of this research was to evaluate the possibility of using a coding approach that contemplated the construction of words using characters as basic tokens. Our main contribution is demonstrate that such approach allows reducing the dimensionality of the encoding matrix, thus allowing substantially shorter optimization times and the use of devices with lower computational power. Some datasets of text have peculiarities that were not addressed by word frequency methods (i.e. BoW and word2vec), like declensions and new vocabulary. The article of was a great innovation in this regard. However, the training times are still obstacles to the most effective use of the technique. We thought of a better representation should be a solution. To make a comparison, perform the training of architecture CNN1 with an output of 4 classes make necessary to optimize 1,837,188 parameters. As a comparison, in the architecture suggested by it is necessary to optimize 11,337,988 parameters. Figure 3: Performance re-scaled in the range between best and worst model comparing and our approaches when applied to the AG's news (AG), Sogou news (SOGOU), DBpedia (DBP), Yelp polarity (YLP-P), Yelp full (YLP), Yahoo! answers (YAH), Amazon full (AMZ) and Amazon polarity (AMZ-P) datasets. As one of our concerns was to make our proposal as usable as possible on commodity hardware, we focused our studies in that hardware configuration. The major bottleneck for analyzing this large amount of matrix-encoded text is the need for intensive use of RAM. Our approach generates a 128 × 256 matrix, smaller than the 1014 × 69 generated by. In spite of that, a large set of them quickly occupies the available RAM on a'regular' personal computer. On the machine we used, there were 16 GB available, which is not uncommon in modern personal computers. Therefore, the use of generators to control the number of matrices generated and sent to GPU is an important detail in the implementation of this optimization algorithm. If your computer has only 8 GB of RAM or less, it will be necessary to reduce the number of superbatchs to fit the memory. The obtained strongly indicate that the use of this coding is a possibility. We emphasize that we used is a fairly simple network, enough to demonstrate the feasibility of the encoding approach with the time and computational resources that we have. The are very competitive with the approach o and traditional techniques. We see even that we could find parameters that achieve excellent performance on AG's news dataset, following the suggestions of BID29. One of advantage of a faster algorithm is that if your dataset is not so big, you could scan the feature width to find a solution that optimizes accuracy. Another advantage is the possibility to realize k-folds validation, to have a better perspective on how well it will perform for your specific dataset on real life. Another interesting point is that using a LSTM layer with the proposed encoding, we achieved similar, sometimes better than using an embedding using word2vec BID20 as proposed by. Our approach by its own nature take into account morphological aspects of text while word2vec uses pre-trained vectors representing co-ocurrency of words on a big corpus of text. Being able to take into account character information even in recurrent networks, we show that this representation is not limited to the domain of CNN or neural networks, for that matter. Although our LSTM architecture is twice slower than our CNN1 and CNN2 topologies, it consistently outperform them. This indicates that temporal dependence among of words are important, so, potentially other architectures can generate better taking this information to account and this is a direction that should be explored. In addition to that, the dimensionality reduction achieved by our encoding enables several other architectures and methods to be verified in a reasonable timeframe. We are certain that our algorithm implementation could be even faster. For instance, when using a GPU Geforce 1080ti and a CNN1 architecture, each of the superbatch of 10,000 arrays have its weights updated in 30 seconds. Only 6 seconds is consumed by GPU, another 24 seconds is spent in encoding all the matrix and delivery it on the GPU. Using a multithread strategy could help in this regard. In this paper, we have proposed an efficient character-level encoding for text derived from the Tagged Huffman BID23 information compression method and applied it as input preprocessing step for character-level CNN text classification. We have shown that using this compression technique is a convenient possibility to represent text in a convolutional deep learning setup for text classification. This is particularly important as encoding text using characters can be relevant when dealing with less curated texts datasets, as it is robust to spelling errors, typos, slangs, language variations, and others usual characteristics of Internet texts. This novel text encoding allow to represent larger portions of texts in a compact form while preserving the ability to encode any word, even those not present in the training dataset ones. Furthermore, for being compact yet sparse, this approach dramatically reduces the time required for training CNNs and, therefore, makes the application of this novel approach more accessible. This opens the door to more complex applications, the use of devices with lower computational power and the exploration of other approaches that can be coupled with this input representation. The experimental studies carried out coupled the encoding with two convolutional neural networks architectures and a recurrent LSTM architecture. These experiments showed that we managed to achieve a performance similar to the one achieved by in a fraction of the training time even if we employed a simpler hardware setup. Furthermore, with our , we endorse , which state that language can be treated as a signal, no different from any other. It should be noted that the main objective of the paper was to show the viability of the text encoding, not producing better per se. Consequently, we have focused our efforts on the comparative study. Probably, custom neural network architectures should be devised to this new encoding with that purpose. Our indicate that combining it with LSTMs is a promising direction in order to overcome the fixed matrix size limitation. In the near future, we will focus on devising these new architectures that may further improve the . This study also opens a door to other information-theoretic-based methods for information representation to be used to create a numerical representation of texts. It must be outlined the fact that this compact numeric representation of text is not limited to the domain of CNN or neural networks, for that matter. It could be interesting to assess its impact as a preprocessing step, perhaps with a minor modification, for other classification algorithms.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Using Compressing tecniques to Encoding of Words is a possibility for faster training of CNN and dimensionality reduction of representation

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: Sequence prediction models can be learned from example sequences with a variety of training algorithms. Maximum likelihood learning is simple and efficient, yet can suffer from compounding error at test time. Reinforcement learning such as policy gradient addresses the issue but can have prohibitively poor exploration efficiency. A rich set of other algorithms, such as data noising, RAML, and softmax policy gradient, have also been developed from different perspectives. In this paper, we present a formalism of entropy regularized policy optimization, and show that the apparently distinct algorithms, including MLE, can be reformulated as special instances of the formulation. The difference between them is characterized by the reward function and two weight hyperparameters. The unifying interpretation enables us to systematically compare the algorithms side-by-side, and gain new insights into the trade-offs of the algorithm design. The new perspective also leads to an improved approach that dynamically interpolates among the family of algorithms, and learns the model in a scheduled way. Experiments on machine translation, text summarization, and game imitation learning demonstrate superiority of the proposed approach. Sequence prediction problem is ubiquitous in many applications, such as generating a sequence of words for machine translation ), text summarization , and image captioning ), or taking a sequence of actions to complete a task. In these problems (e.g., ;), we are often given a set of sequence examples, from which we want to learn a model that sequentially makes the next prediction (e.g., generating the next token) given the current state (e.g., the previous tokens). A standard training algorithm is based on supervised learning which seeks to maximize the loglikelihood of example sequences (i.e., maximum likelihood estimation, MLE). Despite the computational simplicity and efficiency, MLE training can suffer from compounding error in that mistakes at test time accumulate along the way and lead to states far from the training data. Another line of approaches overcome the training/test discrepancy issue by resorting to the reinforcement learning (RL) techniques . For example, used policy gradient to train a text generation model with the task metric (e.g., BLEU) as reward. However, RL-based approaches can face challenges of prohibitively poor sample efficiency and high variance. To this end, a diverse set of methods has been developed that is in a middle ground between the two paradigms of MLE and RL. For example, RAML adds reward-aware perturbation to the MLE data examples; SPG leverages reward distribution for effective sampling of policy gradient. Other approaches such as data noising also show improved . In this paper, we establish a unifying perspective of the above distinct learning algorithms. Specifically, we present a generalized entropy regularized policy optimization framework, and show that the diverse algorithms, such as MLE, RAML, data noising, and SPG, can all be re-formulated as special cases of the framework, with the only difference being the choice of reward and the values of two weight hyperparameters (Figure 1). In particular, we show MLE is equivalent to using a Delta-function reward which returns 1 to model samples that match training examples exactly, and −∞ to any other samples. Such extremely restricted reward has literally disabled any exploration of the model beyond training data, yielding brittle prediction behaviors. Other algorithms essentially use various locally-relaxed rewards, joint with the model distribution, for broader (and more costly) exploration during training. Besides the new views of the existing algorithms, the unifying perspective also leads to new algorithms for improved learning. We develop interpolation algorithm, which, as training proceeds, gradually expands the exploration space by annealing both the reward function and the weight hyperparameters. The annealing in effect dynamically interpolates among the existing algorithms from left to right in Figure 1. We conduct experiments on the tasks of text generation including machine translation and text summarization, and game imitation learning. The interpolation algorithm shows superior performance over various previous methods. Given a set of data examples, sequence prediction models are usually trained to maximize the loglikelihood of the next label (token, action) conditioning on the current state observed in the data. Reinforcement learning (RL) addresses the discrepancy between training and test by also using models' own predictions at training time. Various RL approaches have been applied for sequence generation, such as policy gradient and actor-critic. Reward augmented maximum likelihood (RAML) is an algorithm in between MLE and policy gradient. Mathematically, RAML shows that MLE and maximum-entropy policy gradient are respectively minimizing KL divergences in opposite directions. thus propose to use the more general α-divergence as a combination of the two paradigms. Our framework is developed in a different perspective, reformulates a different and more comprehensive set of algorithms, and leads to new insights in terms of exploration and learning efficiency of the various algorithms. Besides the algorithms discussed in the paper, there are other learning methods for sequence models. For example, Hal Daumé et al.;; use a learning-to-search paradigm for sequence generation or structured prediction. Scheduled Sampling and variants adapt MLE by randomly replacing ground-truth tokens with model predictions as the input for decoding the next-step token. Policy optimization for reinforcement learning is studied extensively in robotic and game environment. For example, introduce a relative entropy regularization to reduce information loss during learning. develop a trust-region approach for monotonic improvement.;; study the policy optimization algorithms in a probabilistic inference perspective. combine imitation learning with RL, whose approach is orthogonal to ours and can be plugged into our framework to incorporate imitation reward. The entropy-regularized policy optimization formulation presented here can be seen as a generalization of many of the previous policy optimization methods. Besides, we formulate the framework primarily in the sequence generation context. We first present a generalized formalism of entropy regularized policy optimization. The formulation contains a reward function and two weight hyperparameters that define the learning procedure. Therefore, varying the values of the reward and weights in a large space of algorithms. We show that several existing popular algorithms, which were originally proposed in distinct perspectives, can all be seen as members in the space. In particular, we reformulate the MLE algorithm in the same policy optimization form, which enables side-by-side comparison between the broad spectrum of algorithms. The ing unifying view provides new insights into the exploration and computation efficiency, and creates improved learning approaches for sequence prediction. For clarity, we present the framework in the sequence generation context. The formulations can straightforwardly be extended to other settings such as imitation learning in robotic and game environments, as discussed briefly at the end of this section and also shown in the experiment. We first establish the basic notations. Let y = (y 1, . . ., y T) be the sequence of T tokens. Let y * be a training example drawn from the empirical data distribution. From the sequence examples, we aim to learn a sequence generation model p θ (y) = t p θ (y t |y 1:t−1) with parameters θ. Note that generation of y can condition on other factors. For example, in machine translation, y is the sentence in target language and depends on an input sentence in source language. For simplicity of notations, we omit the conditioning factors. Policy optimization is a family of reinforcement learning (RL) algorithms. Assume a reward function R(y|y *) ∈ R that evaluates the quality of generation y against the true y *. For example, BLEU score can be a reward in machine translation. The general goal of policy optimization is to learn the model p θ (y) (a.k.a policy) 1 to maximize the expected reward. Previous work develops entropy regularized approaches, which augment the objective with information theoretic regularizers for stabilized training. We present a generalized variational formulation of ERPO, which, as we show shortly, has the power of subsuming an array of other popular algorithms. Specifically, we introduce a non-parametric variational distribution q(y) w.r.t the model p θ (y). The objective to maximize is as follows: where KL(· ·) is the Kullback-Leibler divergence forcing q to stay close to p θ; H(·) is the Shannon entropy imposing maximum entropy assumption on q; and α and β are balancing weights of the respective terms. Intuitively, the objective is to maximize the expected reward under the variational distribution q while minimizing the distance between q and the model p θ, with maximum entropy regularization on q. The above formulation is relevant to and can be seen as a variant of previous policy optimization approaches in RL literature, such as relative entropy policy search , maximum entropy policy gradient , and other work where the variational distribution q is formulated either as a non-parametric distribution as ours or parametric one (; 2017a;). The objective can be maximized with a standard EM procedure that iterates two coordinate ascent steps optimizing q and θ, respectively. At iteration n: In the E-step, q has a closed-form solution, which is an energy-based distribution. We can have an intuitive interpretation of its form. First, it is clear to see that if α → ∞, we have q n+1 = p n θ. This is also reflected in the objective Eq. where a larger weight α encourages q to be close to p θ. Second, the weight β serves as the temperature of the q softmax distribution. In particular, a large temperature β → ∞ makes q a uniform distribution, which is consistent with the outcome of an infinitely large maximum entropy regularization in Eq.. In the M-step, the update rule can be interpreted as maximizing the log-likelihood of samples from the distribution q. In the context of sequence generation, it is sometimes more convenient to express the equations at token level (instead of the sequence level), as shown when we devise a new algorithm in the next section. To this end, we decompose R(y|y *) along the time steps: where ∆R(y t |y *, y 1:t−1) measures the reward contributed by token y t. The solution of q in Eq. can then be re-written as: The Algorithm Space The above ERPO formalism includes three key components, namely, the reward R and the weight hyperparameters α and β > 0. Variation in these components can in different procedures of updating the model. In other words, different algorithms in the ERPO family correspond to a point (or a region) in the space spanned by the three components. The following sections visit a set of existing approaches, and connect them to the unifying picture by reformulating their seemingly distinct objectives. Figure 1 illustrates the particular algorithms in the space, clustered by the exploration behavior in learning, of which we will discuss more. Softmax Policy Gradient (SPG) We first briefly discuss the previous RL algorithms for sequence prediction that fit in the ERPO formalism. SPG was originally developed in the perspective of combining the reward R and policy p θ to improve sampling quality. The algorithm is equivalent to setting β = 0 and treating α > 0 as the temperature of the energy-based distribution q(y). That is, q(y) in the E-step of Eq. is now in the form q(y) ∝ p θ (y) exp{R(y|y *)/α}. The reward R is set to any normal task-specific reward. Note that sampling from q(y) (e.g., in the M-step) is typically difficult due to its energy-based form and the fact that the task reward R often does not have particular structures amenable for sampling. We will see in the next section that the MLE algorithm in contrast uses a special reward to avoid the computational difficulty in sampling, at the cost of restricted exploration during training. We also note the previous work of Sequence Tutor , which was motivated by the idea of using an MLE-trained policy as a prior to guide the learning of the target policy in an RL framework. The formalism closely resembles SPG, namely (α > 0, β = 0), with the exception that the variational distribution q(y) in Sequence Tutor is a parameterized model instead of a nonparametric one as in SPG and our more general ERPO formulation. In this section, we connect the maximum likelihood estimation (MLE) algorithm to the unifying ERPO formalism. Based on the connections, we are able to analyze the learning behavior of MLE from the reinforcement learning perspective in terms of exploration efficiency. We also discuss some well-known variants of the vanilla MLE algorithm, such as RAML and data augmentation. Due to its simplicity and efficiency, MLE is among the most widely-used approaches in learning sequence generation. It finds the optimal parameter value that maximizes the data log-likelihood: We show that the MLE objective can be recovered from Eq. with a specialized reward and weight values. More concretely, consider a δ-function reward defined as 2: That is, a sample y receives a valid unit reward only when it matches exactly with the true data, and receives a negative infinite reward in all other cases. We show that the MLE algorithm is a member of the ERPO family. In particular, the conventional MLE objective is equivalent to setting the ERPO components to (R = R δ, α → 0, β = 1). This can 2 For token-level, define R δ (y1:t|y *) = t/T * if y1:t = y * 1:t and −∞ otherwise, where T * is the length of y *. Note that the R δ value of y = y * can also be set to any constant larger than −∞. be straightforwardly seen by noting that, with the configuration, the q(y) in E-step (Eq.2) reduces to q(y) = 1 if y = y * and 0 otherwise. The M-step is thus in effect maximizing the log-likelihood of the real data examples (Note that the very small α is still > 0, making the M-step for maximizing the objective Eq. valid and necessary). With the δ-reward R δ, any sample y that fails to match the given data y * exactly will get a negative infinite reward and thus never contribute to model learning. Exploration Efficiency Reformulating MLE in the unifying ERPO form enables us to directly compare the approach with other RL algorithms. Specifically, the δ-reward has permitted only samples that match training examples, and made invalid any exploration beyond the small set of training data (Figure 2(a) ). The extremely restricted exploration at training time in a brittle model that can easily encounter unseen states and make mistakes in prediction. On the other hand, however, a major advantage of the δ-reward is that it defines a distribution over the sequence space such that sampling from the distribution is reduced to simply picking an instance from the training set. The ing samples are ensured to have high quality. This makes the MLE implementation very simple and the computation efficient in practice. On the contrary, task-specific rewards (such as BLEU) used in standard policy optimization are more diffused than the δ-reward, and thus allow exploration in a broader space with valid reward signals. However, the diffused rewards often do not lead to a distribution that is amenable for sampling as above. The model distribution is thus instead used to propose samples, which in turn can yield low-quality (i.e., low-reward) samples especially due to the huge sequence space. This makes the exploration inefficient or even impractical. Given the opposite behaviors of the algorithms in terms of exploration and computation efficiency, it is a natural idea to seek a middle ground between the two extremes in order to combine the advantages of both. Previous work has proposed variants of the vanilla MLE from different perspectives. We re-visit some of the popular approaches, and show that they can also be canonicalized in the ERPO framework and enrich our understanding of the learning behaviors. Data Noising Adding noise to training data is a widely adopted model regularizing technique. Previous work (e.g.,) has proposed several data noising strategies in the sequence generation context, such as replacing subsets of tokens with other random words. The ing noisy data is then used in MLE training. Though previous literature has commonly seen such techniques as a data pre-processing step, we show that the approach can be expressed in the generalized ERPO formulation. Specifically, data noising can be seen as using a locally relaxed variant of the δ-reward: where g denotes any transformation operation that returns a new sample as a noisy version of the input raw data y *. With the relaxed reward, data noising locally expands the exploration surrounding the observed training examples (Figure 2(b) ). The added exploration at training time can yield a model that is more robust to error at test time. Reward-Augmented Maximum Likelihood (RAML) RAML was originally proposed to incorporate task-specific metrics into the MLE training. Formally, it introduces an exponentiated reward distribution e(y|y *) ∝ exp{R(y|y *)/τ }, where R is a task reward and τ > 0 is the temperature. The conventional RAML objective is written as: That is, unlike MLE that directly maximizes the data log-likelihood, RAML first perturbs the data proportionally to the reward distribution, and maximizes the log-likelihood of the ing samples. Similar to how we map MLE to the ERPO formalism, we can align RAML with the unifying form by setting α → 0, β to the temperature τ, and R to the task reward. Compared to the vanilla MLE, the key feature of RAML is the use of task reward instead of the δ-reward, which permits a larger exploration space surrounding the training examples. On the other hand, same as in SPG (section 3.1), sampling from the energy-based distribution with a diffused reward tends to be difficult, and often requires specialized approximations for computational efficiency (e.g.,). Other Algorithms & Discussions The classic policy gradient algorithm has also been used for sequence prediction (e.g.,). We We show in the appendix that the approach can also be connected to the unifying ERPO with moderate approximations. also proposed a mixing training strategy that anneals from MLE training to policy optimization. We show in the next section that the particular annealing scheme is a special case of the new, more general interpolation algorithm below. We have presented the framework in the context of sequence generation. The formulation can also be extended to other settings. For example, in game environments, y is a sequence of actions and states. The popular imitation learning method GAIL uses an adversarially induced reward R from data, and applies standard RL updates to train the policy. The policy update part can be formulated with our framework as standard policy optimization (with α > 0, β = 0). The new interpolation algorithm described in the next section can also be applied to improve the vanilla GAIL, as shown in the experiments. Previous work has also studied connections of relevant algorithms. For example,; formulate MLE and policy gradient as minimizing the opposite KL divergences between the model and data/reward distributions. studied an update equation generalizing maximum marginal likelihood and policy gradient. Our framework differs in that we reformulate a different and more comprehensive set of algorithms for sequence prediction, and provide new insights in terms of exploration and its efficiency, which could not be derived from the previous work. Section 2 discusses more related work on sequence prediction learning. The unifying perspective also leads to new algorithms for improved learning. Here, we present an example algorithm that is naturally inspired by the framework. As in Figure 1, each of the learning algorithms can be seen as a point in the (R, α, β) space. Generally, from left to right, the reward gets more diffused and α gets larger, which in larger sequence space exposed to model training (Figure 2). More exploration in turn also makes the training less efficient due to lower sample quality. We propose an interpolation algorithm with the natural idea of starting learning from the most restricted yet efficient algorithm configuration, and gradually expanding the exploration to decrease the training/test discrepancy. The easy-to-hard learning paradigm resembles the curriculum learning . As we have mapped the algorithms to the points in the hyperparameter space, the interpolation becomes straightforward, which reduces to simple annealing of the hyperparameter values. Specifically, during training, we would like to anneal from using the restricted δ-reward R δ to using task reward, and anneal from sampling (exploring) by only the reward R to sampling by both R and p θ. Since R δ is a δ-function which would make direct function linear combination problematic, we implement the interpolation strategy in the update rule (Eq.2) and use log-sum-exp for mixing. Formally, let R task denote a task reward. The negative energy of q(y) in Eq. (i.e., the exponent inside exp{·}) is now replaced with the interpolated term: log(λ 1 p θ +λ 2 exp{R task}+λ 3 exp{R δ}). Note that we have re-organized the weight hyperparameters and used the distribution (λ 1, λ 2, λ 3) to carry out the calibration role of (α, β). In particular, as training proceeds, we gradually increase λ 1 and λ 2 and decrease λ 3. The formulation of interpolation in effect converts the energy-based model q(y) to a mixture of experts, which makes the sampling from q(y) easier, and resembles the Table 2: Text summarization (5-run average ± std dev). bang-bang rewarded SPG method as described in . Besides, similar to , we adopt the token-level formulation (Eq.4), so that tokens in a sequence can be sampled from different components (i.e., p θ, R task, and R δ) in a mixed way. We provide the pseudo-code of the interpolation algorithm in the appendix. As discussed above, we can also apply the interpolation algorithm in game imitation learning, by plugging it into the GAIL framework to replace the standard RL routine for policy update. The annealing schedule in this setting is constrained due to the agent interaction with the environment. Specifically, to generate a trajectory (a sequence of actions and states), we sample the beginning part from data (demonstrations), followed by sampling from either the model or reward. Note that data sampling can happen only before model/reward sampling, because the latter will interact with the environment and in states that do not necessarily match the data. Similar to sequence generation, we gradually anneal from data sampling to model/reward sampling, and hence increase the exploration until converging to standard RL. Our experiments validate that the easy-to-hard training is superior to the vanilla GAIL which directly applies the hard RL update from the beginning. It is notable that also developed an annealing strategy that mixes MLE and policy gradient training. The strategy is essentially the same as the one we apply in the GAIL learning setting. That is, the annealing approach of is a specialized case of the above more general annealing, using restricted values of (λ 1, λ 2, λ 3) and discrete changes. We provide more discussions in the appendix. The experiment in section 5 show that our generalized annealing performs better than the restricted approach . We evaluate the interpolation algorithm in the context of both text generation and game imitation learning. Experiments are run with 4 GTX 2080Ti GPUs and 32GB RAM. The link to the code is provided in the submission. We will release the code upon acceptance. We use the state-of-the-art neural architecture Transformer as the base model. The model has 6 blocks, trained with an Adam optimizer with an initial learning rate of 0.001 and the same schedule as in . Batch size is 1,792 tokens. At test time, we use beam search decoding with a beam width of 5 and length penalty 0.6. We use the popular IWSLT2014 German-English dataset. After proper pre-processing as described in the appendix, we obtain the final dataset with train/dev/test size of around 146K/7K/7K, respectively. The shared de-en vocabulary is of size 73,197 without BPE encoding. Table 1 shows the test-set BLEU scores of various methods. Besides MLE, RAML, and MIXER as discussed above, we also compare with other existing approaches such as Scheduled Sampling (SS) and Self-critic . (We did not compare with SPG as no public code is available.) From the table, we can see the various approaches provide improved performance over the vanilla MLE, as more sufficient exploration is made at training time. Our interpolation algorithm performs best, with significant improvement over the MLE training by 1.36 BLEU points. The validate our approach that interpolates among the existing algorithms offers beneficial scheduled training. To further study the effect of our generalized annealing versus the MIXER strategy, we compare with "MIXER-alike Aneal" which uses the same configuration with our interpolation algorithm, except that the annealing is restricted like MIXER. That is, the first portion of tokens in a sequence are all sampled from the data, while the subsequent tokens are sampled from only the model or the task reward. We see that the proposed more generalized annealing is superior to the restricted version. We note that there is other work exploring various network architectures for machine translation (;, which is orthogonal and complementary to the learning algorithms. It would be interesting to explore the effect of combining the approaches. We use an attentional sequence-to-sequence model where both the encoder and decoder are single-layer LSTM RNN. The dimensions of word embedding, RNN hidden state, and attention are all set to 256. We use Adam optimization with an initial learning rate of 0.001 and a batch size of 64. Test time uses beam search decoding with a beam width of 5. Please see the appendix for more configuration details. We use the popular English Gigaword corpus for text summarization, and pre-processed the data following . The ing dataset consists of 200K/8K/2K source-target pairs in train/dev/test sets, respectively. Following previous work , we use the summation of the three ROUGE(-1, -2, -L) metrics as the reward in learning. Table 2 show the on the test set. The proposed interpolation algorithm achieves the best performance on all three metrics. The RAML algorithm, which performed well in machine translation, falls behind other algorithms in text summarization. In contrast, our method consistently provides the best . We apply the interpolation algorithm in GAIL as described in section 4. Following , we simulate three environments with MuJoCo . Expert demonstrations are generated by running PPO (b) under the given true reward functions. We then run different imitation learning algorithms with varying numbers of demonstrations. Both the policy and the discriminator are two-layer networks with 128 units each and tanh activations in between. Figure 3 shows the average returns by the agents. We can see that agents trained with the interpolation algorithm can generally improve over the vanilla GAIL, especially in the presence of small number (e.g., 1 or 4) of demonstrations. This shows that our approach that anneals from the MLE mode to RL mode can make better use of data examples, and steadily achieve better performance in the end. We present the learning curves of the algorithms in the appendix. We have presented a unifying perspective of a variety of learning algorithms for sequence prediction problems. The framework is based on a generalized entropy regularized policy optimization formulation, and we show the distinct algorithms are equivalent to specifying the reward and weight hyperparameters. The new consistent treatment provides systematic understanding and comparison across the algorithms, and inspires further improved learning. The proposed interpolation algorithm shows consistent improvement in machine translation, text summarization, and game imitation learning. made an early attempt to address the exposure bias problem by exploiting the policy gradient algorithm . Policy gradient aims to maximizes the expected reward: where RP G is usually a common reward function (e.g., BLEU). Taking gradient w.r.t θ gives: We now reveal the relation between the ERPO framework we present and the policy gradient algorithm. Starting from the M-step of Eq. and setting (α = 1, β = 0) as in SPG (section ??), we use p θ n as the proposal distribution and obtain the importance sampling estimate of the gradient (we omit the superscript n for notation simplicity): Eq [∇ θ log p θ (y)] = Ep θ q(y) p θ (y) ∇ θ log p θ (y) = 1/Z θ · Ep θ exp{R(y|y *)} · ∇ θ log p θ (y), where Z θ = y exp{log p θ + R} is the normalization constant of q, which can be considered as adjusting the step size of gradient descent. We can see that Eq. recovers Eq. if we further set R = log RP G, and omit the scaling factor Z θ. In other words, policy gradient can be seen as a special instance of the general ERPO framework with (R = log RP G, α = 1, β = 0) and with Z θ omitted. The MIXER algorithm incorporates an annealing strategy that mixes between MLE and policy gradient training. Specifically, given a ground-truth example y *, the first m tokens y * 1:m are used for evaluating MLE loss, and starting from step m + 1, policy gradient objective is used. The m value decreases as training proceeds. With the relation between policy gradient and ERPO as established above, MIXER can be seen as a specific instance of the proposed interpolation algorithm (section 4) that follows a restricted annealing strategy for token-level hyperparameters (λ1, λ2, λ3). That is, for t < m in Eq.4 (i.e.,the first m steps), (λ1, λ2, λ3) is set to and c = 1, namely the MLE training; while for t > m, (λ1, λ2, λ3) is set to (0.5, 0.5, 0) and c = 2. Algorithm 1 summarizes the interpolation algorithm described in section 4. A.3.1 DATA PRE-PROCESSING For the machine translation dataset, we follow for data pre-processing. In text summarization, we sampled 200K out of the 3.8M pre-processed training examples provided by for the sake of training efficiency. We used the refined validation and test sets provided by . In the game imitation learning task, we randomly sample 50 state-action pairs in each trajectory as demonstrations. Every training iteration, we collect at least 2,048 state-action pairs, and we train 1,000 iterations for every model in every environment.

An entropy regularized policy optimization formalism subsumes a set of sequence prediction learning algorithms. A new interpolation algorithm with improved results on text generation and game imitation learning.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Parametric adversarial divergences implicitly define more meaningful task losses for generative modeling, we make parallels with structured prediction to study the properties of these divergences and their ability to encode the task of interest.


Question: Represent the scientific article to find a short TLDR

Question: Origin-Destination (OD) flow data is an important instrument in transportation studies. Precise prediction of customer demands from each original location to a destination given a series of previous snapshots helps ride-sharing platforms to better understand their market mechanism. However, most existing prediction methods ignore the network structure of OD flow data and fail to utilize the topological dependencies among related OD pairs. In this paper, we propose a latent spatial-temporal origin-destination (LSTOD) model, with a novel convolutional neural network (CNN) filter to learn the spatial features of OD pairs from a graph perspective and an attention structure to capture their long-term periodicity. Experiments on a real customer request dataset with available OD information from a ride-sharing platform demonstrate the advantage of LSTOD in achieving at least 6.5% improvement in prediction accuracy over the second best model. Spatial-temporal prediction of large-scale network-based OD flow data plays an important role in traffic flow control, urban routes planning, infrastructure construction, and the policy design of ridesharing platforms, among others. On ride-sharing platforms, customers keep sending requests with origins and destinations at each moment. Knowing the exact original location and destination of each future trip allows platforms to prepare sufficient supplies in advance to optimize resource utilization and improve users' experience. Given the destinations of prospective demands, platforms can predict the number of drivers transferring from busy to idle status. Prediction of dynamic demand flow data helps ride-sharing platforms to design better order dispatch and fleet management policies for achieving the demand-supply equilibrium as well as decreased passenger waiting times and increased driver serving rates. Many efforts have been devoted to developing traffic flow prediction models in the past few decades. Before the rise of deep learning, traditional statistical and machine learning approaches dominate this field (; ; ; ; Idé & ;). Most of these models are linear and thus ignore some important non-linear correlations among the OD flows. Some other methods further use additional manually extracted external features, but they fail to automatically extract the spatial representation of OD data. Moreover, they roughly combine the spatial and temporal features when fitting the prediction model instead of dynamically modelling their interactions. The development of deep learning technologies brings a significant improvement of OD flow prediction by extracting non-linear latent structures that cannot be easily covered by feature engineering. (; ;). Zhang et al. (2016; modeled the whole city are as an entire image and employed residual neural network to capture temporal closeness. and also learned traffic as images but they used LSTM instead to obtain the temporal dependency. Yao et al. (2018b) proposed a Deep Multi-View Spatial-Temporal Network (DMVST-Net) framework to model both spatial and temporal relationships. However, using standard convolution filters suffers from the problem that some OD flows covered by a receptive field of regular CNNs are not spatially important. Graph-based neural net-works (GNN) (; ; Veličković et al., 2017) are proved to be powerful tools in modelling spatial-temporal network structures ). However, none of these frameworks are directly applicable here since both the historical observations and responses to predict are vertex-level variables. On the contrary, the OD flows we discuss in this paper are generated in the edge space by our definition. The aim of this paper is to introduce a hierarchical Latent Spatial-Temporal Origin-Destination (LSTOD) prediction model to jointly extract the complex spatial-temporal features of OD data by using some well-designed CNN-based architectures. Instead of modelling the dynamic OD networks as a sequence of images and applying standard convolution filters to capture their spatial information, we introduce a novel Vertex Adjacent Convolution Network (VACN) that uses an irregular convolution filter to cover the most related OD flows that share common vertecies with the target one. The OD flows connected by common starting and/or ending vertexes, which may fall into different regions of the flow map, can be spatially correlated and topologically connected. Moreover, for most ride-sharing platforms, a passenger is more likely to send a new request from the location where his/her last trip ends in. To learn such sequential dependency, we introduce a temporal gated CNN (TGCNN) and integrate it with VACN by using the sandwich-structured STconv block in order to collectively catch the evolutionary mechanism of dynamic OD flow systems. A periodically shifted attention mechanism is also used to capture the shift in the long-term periodicity. Finally, the combined short-term and long-term representations are fed into the final prediction layer to complete the architecture. Our contributions are summarized as follow: • To the best of our knowledge, it is the first time that we propose purely convolutional structures to learn both short-term and long-term spatio-temporal features simultaneously from dynamic origin-destination flow data. • We propose a novel VACN architecture to capture the graph-based semantic connections and functional similarities among correlated OD flows by modeling each OD flow map as an adjacency matrix. • We design a periodically shift attention mechanism to model the long-term periodicity when using convolutional architecture TGCNN in learning temporal features. • Experimental on two real customer demand data sets obtained from a ride-sharing platform demonstrate that LSTOD outperforms many state-of-the-art methods in OD flow prediction, with 7.94% to 15.14% improvement of testing RMSE. For a given urban area, we observe a sequence of adjacency matrices representing the OD flow maps defined on a fixed vertex set V, which indicates the N selected sub-regions from this area. We let V = {v 1, v 2, . . ., v N} denote the vertex set with v i being the i-th sub-region. The shape of each grid v i could be either rectangles, hexagons or irregular sub-regions. We define the dynamic OD flow maps as {O The goal of our prediction problem is to predict the snapshot O d,t+j ∈ R N ×N in the future time window (t + j) of day d given previously observed data, including both short-term and long-term historical information. The short-term input data consists of the last p 1 timestamps from t + 1 − p 1 to t, denoted by The long-term input data is made up of q time series {O d−ϕ,t+j−(p2−1)/2,..., O d−ϕ,t+j+(p2−1)/2 } of length p 2 for each previous day (d − ϕ) with the predicted time index (t + j) in the middle for ϕ = 1,..., q. We let,t+j+(p2−1)/2 } denote the entire long-term data. Increasing p 1 and p 2 leads to higher prediction accuracy, but more training time. We reformulate the set of short-term OD networks X 1 into a 4D tensor X 1 ∈ R N ×N ×p1×1 and concatenate the long-term snapshots X 2 into a 5D tensor Each X 2,d−ϕ ∈ R N ×N ×p2×1 for day d − ϕ is a 4D tensor for ϕ = 1,..., q. Therefore, we can finally define our latent prediction problem as follows: where F (·, ·) represents the LSTOD model, which captures the network structures of OD flow data as well as the temporal dependencies in multiple scales. A notation table is attached in the appendix. In this section, we describe the details of our proposed LSTOD prediction model. See Figure 1 for the architecture of LSTOD. The four major novelties and functionalities of LSTOD model include • an end-to-end framework LSTOD constructed by all kinds of CNN modules to process dynamic OD flow maps and build spatio-temporal prediction models; • a novel multi-layer architecture VACN to extract the network patterns of OD flow maps by propagating through edge connections, which can not be covered by traditional CNNs; • a special module ST-Conv block used to combine VACN and gated temporal convolutions to coherently learning the essential spatio-temporal representations; • a periodically shifted attention mechanism which is well designed for the purely convolutional ST-Conv blocks to efficiently utilize the long-term information by measuring its similarities with short-term data. Before introducing the detailed architecture of VACN, we want to discuss why directly applying standard CNNs to the OD flow map O d,t may disregard the connections between neighboring OD flows in the graph space first. Figure 2 demonstrates that it fails to capture enough semantic information using the real-world example of ride demands. For the OD flow starting from v 1 to v 2, as illustrated in the upper sub-figure, the most related OD flows should be those with either origin or destination being v 1 or v 2 in the past few timestamps. A certain part of the travel requests from v 1 to v 2 can be matched with some historical finished trips from a third-party location to V 1 by the same group of people, for example a trip from v 3 to v 1. However, as the lower-left sub-figure illustrates, some of the OD flows covered by a single CNN filter (the green square) such as the four corners of the kernel window may be topologically far away from the target one in the graph. More generally, let's consider a target OD flow o To better understand the differences between our proposed VACN over graph edges with those vertex-based convolutions such as GCN or GAT (Veličković et al., 2017), we introduce the concept of line graphs L(G) . Each node in L(G) corresponds to an edge in G, while each individual edge in L(G) can be mapped to a pair of edges in G that connect to a joint vertex. L(G) contains representing a feature vector at the edge from v i to v j. The learned representation of each target edge is defined as the weighted sum of those from the same row or column in the adjacency matrix, and those from the row or column in the transposed adjacency matrix. The output of the layer-wise VACN propagation for the OD flow from v i to v j is defined as follows: where are the weights to learn. F(·) represents an elementwise activation function, such as ReLU(x) = max(0, x). The first part of We use temporal gated CNN (TGCNN) instead of RNN-based architectures such as LSTMs to capture the temporal representation, which makes our LSTOD a pure convolutional architecture. RNNs suffer from the problem of lower training efficiency, gradient instability, and time-consuming convergence. Moreover, the high dimension of the spatial representations captured by VACN and a potential long temporal sequence length make RNNs notoriously difficult to train. The CNNs is more flexible in handling various data structures and allows parallel computations to increase training speed. TGCNN consists of two parts including one being a 3D convolution kernel applied to the spatial representations of all the N 2 OD flows along the time axis and the other being a gated linear unit (GLU) as the gate mechanism. By employing VACN at each of r successive timeslots, we can build a 4D tensor Y = (Y d,t) ∈ R N ×N ×r×ms which is then fed into the TGCNN operator: where G * γ represents the TGCNN kernel and γ includes all the related parameters to learn. 2m t 3D convolutional kernels of size (1 × 1 × K) with zero paddings map the input Y into a single output, which is split in half to obtain Y 1 and Y 2 with the same number of feature channels. here denotes the element-wise Hadamard product. We use the spatial-temporal convolutional block (ST-conv block) to jointly capture the spatialtemporal features of OD flow data, which has a'sandwich'-structure architecture with a multi-layer VACN operator in the middle connecting the two TGCNNs. The use of ST-Conv blocks have two major advantages. First, the block can be stacked or extended based on the dimension and characteristic of the spatio-temporal input data. Second, a temporal operation is applied before extracting the spatial information, which greatly reduces its computation complexity and memory consumption. Both the input and output of each individual ST-Conv block are 4D tensors. For the input of the l-th block Z l−1 ∈ R N ×N ×r l−1 ×c l−1 (Z 0 is the original the OD flow data with c 0 = 1), the output is computed as follows: where G 1 * γ l 1 and G 0 * γ l 0 are two TGCNN kernels and S * θ l is a multi-layer VACN operator being applied to each timestamp. The G 1 and G 2 operators from all the stacked ST-Conv blocks employ the same kernel sizes, which are (1 × 1 × K 1) and (1 × 1 × K 2), respectively. Thus, we have r l = r l−1 − (K 1 + K 2 − 2). After applying (r 0 − 1)/(K 0 + K 1 − 2) ST-Conv blocks to the input Z 0, the temporal length is reduced from r 0 to 1. When input being the short-term OD flow data f here squeezes the captured 4D output into a 3D tensor by dropping the temporal axis. The kernel sizes of each G 1 and, respectively. The detailed propagation of the l-th ST-Conv block is defined as In addition to capturing the the spatial-temporal features from short-term OD flow data X 1, we also take into account the long-term temporal periodicity due to the potential day-wise cycling patterns insides the OD flow data, decided by customer's travelling schedule and the city's traffic conditions. Directly applying ST-Con blocks to an extremely long OD sequence which covers previous few days or weeks is computationally expensive. Only a small set of timestamps from each previous day is necessary to capture the long-term periodicity. As mentioned, we pick p 2 time intervals for each day d − ϕ when predicting the time window (d, t + j) considering the non-strict long-term periodicity. This slight time shifting may be caused by unstable traffic peaks, holidays and extreme weather conditions among different days. Inspired by the recently widely used attention mechanisms (; a;) in spatial-temporal prediction problems, we propose a modified periodically shifted attention to work for the CNN-based ST-Conv blocks here. Different from Yao et al. (2018a) that measures the similarity between hidden units of LSTMs, the attention here is built on the intermediate outputs of TGCNNs where the concatenations are then fed into a new set of ST-Conv blocks. For each day (d − ϕ), we apply a series of L 1 ST-Conv blocks to the day-level p 2 -length sequential OD flow data X 2,d−ϕ and reduce the temporal length from p 2 to n 0 LT. Each block contains two TGCNN layers with the same kernel size 1 and the propagation rule of the l-th ST-Conv blocks is defined as: with Z Moreover, score(z where W 1, W 2 and v φ are learned projection matrices. b s is the added bias term. By assuming that to build a new 4D day-wise time series Z 0 LT . and finally apply another set of ST-Conv blocks to it to obtain the long-term spatial-temporal representations, which is denoted by Z LT ∈ R N ×N ×c LT . c LT is the number of feature channels. We concatenate the short-term and long-term spatial-temporal representations Z ST and Z LT together along the feature axis as Z = Z ST ⊕ Z LT ∈ R N ×N ×C, where C = c ST + c LT . Then, Z is modified to a 2D tensor Z ∈ R N 2 ×C by flattening the first two dimensions while keeping the third one. We apply a fully connected layer to the C feature channels together with an element-wise non-linear sigmoid function to get the final predictions for all the N 2 OD flows. We normalize the original OD flow data in the training set to by Max-Min normalization and use'sigmoid' activation for the final prediction layer to ensure that all predictions fall into. The upper and lower bounds are saved and used to denormalize the predictions of testing data to get the actual flow volumes. We use L 2 loss to build the objective loss during the training. The model is optimized via Backpropagation Through Time (BPTT) and Adam . The whole architecture of our model is realized using Tensorflow and Keras . All experiments were run on a cluster with one NVIDIA 12G-memory Titan GPU. In this section, we compare the proposed LSTOD model with some state-of-the-art approaches for latent traffic flow predictions. All compared methods are classified into traditional statistical methods and deep-learning based approaches. We use the demand flow data collected by a ride-sharing platform to examine the finite sample performance of OD flow predictions for each method. We employ a large-scale demand dataset obtained from a large-scale ride-sharing platform to do all the experiments. The dataset contains all customer requests received by the platform from 04/01/2018 to 06/30/2018 in two big cities A and B. Within each urban area, N = 50 hexagonal regions with the largest customer demands are selected to build the in total N 2 = 2500 OD flows. Since one-layer VACN has a computation complexity O(N) at each of the N 2 entries (globally O(N 3)), the memory consumption highly increases as N gets bigger. Considering the computation efficiency and storage limitation, we choose N = 50 here which can cover more than 80% of total demands and thus satisfy the operation requirement of the ride-sharing platform. We split the whole dataset into two parts. The data from 04/01/2018 to 06/16/2018 is used for model training, while the other part from 06/17/2017 to 06/30/2017 (14 days) serves as the testing set. The first two and half months of OD flow data is further divided in half to the training and validation sets. The size ratio between the two sets is around 4:1. We let 30 min be the length of each timestamp and the value of the OD flow from v i to v j is the cumulative number of customer requests. We make predictions for all the 50 2 OD flows in the incoming 1st, 2nd, and 3rd 30 minutes (i.e. t + 1, t + 2, t + 3) by each compared method, given the historical data with varied (p 1, p 2) combinations. For those model settings incorporating long-term information, we trace back q = 3 days to capture the time periodicity. We use Rooted Mean Square Error to evaluate the performance of each method: All state-of-the-art methods to be compared are listed as follows, some of which are modified to work for the OD flow data: (i) Historical average (HA): HA predicts the demand amount at each OD flow by the average value of the same day in previous 4 weeks. (ii) Autoregressive integrated moving average (ARIMA), (iii) Support Vector Machine Regression (SVMR), (iv) Latent Space Model for Road Networks (LSM-RN) , (v) Dense + BiLSTM (DLSTM) (Altché & de) and (vi) Spatiotemporal Recurrent Convolutional Networks (SRCN). We only consider latent models in this paper, that is, no external covariates are allowed, while only the historical OD flow data is used to extract the hidden spatial-temporal features. We tune the hyperparameters of each compared model to obtain the optimal prediction performance. Specifically, we get (p *, d *, q *) = for ARIMA and k * = 15, γ * = 2 −5, λ * = 10 for LSM-RN. The optimal kernel size of the spatial based CNN kernel is 11 × 11 in SRCN model. For fair comparison, we set the length of short-term OD flow sequence to be p 1 = 9 (i.e., previous 4.5 hours), q = 3 for long-term data which covers the three most recent days, and the length of each day-level time series p 2 = 5 to capture the periodicity shifting (one hour before and after the predicted time index). More analysis of how variant (p 1, p 2) combinations may affect the prediction performance of LSTOD will be studied latter. A two-layer architecture is used by all the deep-learning based methods to extract the spatial patterns inside the OD flow data (L = 2 for both short-term and long-term VACN). We set the filter size of all deep learning layers in both spatial and temporal space to be 64, including the VACNs and TGCNNs in our LSTOD model with c ST = c LT = 64. Comparison with state-of-the-art methods. Table 1 summarizes the finite sample performance for all the competitive methods and our LSTOD model in terms of the prediction RMSE on the testing data of city A. We compute the mean, variance, 25% quantile and 75% quantile of the 14 day-wise RMSE on the testing set. LSTOD outperforms all other methods on the testing data with the lowest average day-wise RMSE (2.41/2.55/2.67), achieving (8.02%/7.94%/8.24%) improvement over the second best method'SRCN'. In general, deep-learning based models perform more stably than traditional methods with smaller variance and narrower confidence intervals. Both'ARIMA' and'LSM-RN' perform poorly, even much worse than HA, indicating that they cannot capture enough short-term spatial-temporal features to get the evolution trend of OD flow data. Among the deep learning models, LSTOD can more efficiently control the estimation variance compared to all the others. This demonstrates the advantages of using our spatial-temporal architecture and long-term periodicity mechanism in modelling the dynamic evolution of OD flow networks. The improvement becomes more significant when the time scale increases since the contributions of long-term periodicity are more emphasized as the short-term signals getting weaker. The LSTOD performs even better on city B compared to the baseline methods since the long-term periodical pattern in city B may be more significant compared with that in city A. Detailed about City B are summarized in Table 3 of the appendix. Comparison with variants of LSTOD. Table 2 shows the finite sample performance of our proposed model LSTOD and its different variants based on the demand data from city A. We can see that the complete LSTOD model outperforms the short-term model and the one without using attention mechanisms in terms of smaller means and variances, and narrower confidence intervals. It indicates that the attention design we use can capture the shift of the day-wise periodicity and extract more seasonal patterns to improve prediction accuracy. The left sub-plot of Figure 3 compares the predictions by each model against the true values at two selected OD flows on the last 3 testing days in the 60-minute scales. Two abnormal change points are marked by black circles. The short-term model fails in this case because it ignores the long-term information. The complete LSTOD model outperforms the one without using attention mechanisms since it can better catch the shift of the periodicity. The right sub-plot visualizes the distribution curves of the day-wise RMSEs on the 14 testing days by each of the three compared models. The lighter tail of the red curve demonstrates that the complete LSTOD is more predictive and stable especially for those unusual cases. We do some more experiments to show how different hyperparameter configurations influence the model performance. For more details, please refer to Section E of the appendix. VACN VS standard local CNN. In this experiment, we will show that our proposed VACN outperforms standard CNNs in capturing the hidden network structure of the OD flow data. Given the model setting that N = 50 sub-regions of city A are used to build the dynamic OD flow matrices, the number of pixels being covered by VACN at each single snapshot is 50 × 4 = 200. For fair comparison, the largest receptive filed of standard CNN should be no bigger than a 15 × 15 window, which includes 225 elements each time. We consider five different kernel sizes including 5 × 5, 8 × 8, 11 × 11, 14 × 14, and 15 × 15. We replace VCAN in our model by standard CNN in order to fairly compare its performance. All hyper-parameters are fixed but only the kernel size of CNNs being changed. Moreover, we only consider the baseline short-term mode of LSTOD model while ignoring the long-term information. As Figure 4 illustrates, standard CNN achieves the best performance with the smallest RMSE = 2.64 on testing data with the filter size being 11 × 11, which is still higher than that using VACN with RMSE = 2.54. Specifically, RMSE increases when the receptive field is larger than 11 × 11 since the spatial correlations among the most related OD flows (sharing common origin or destination nodes) are smoothed with the increase in the filter size ((8 × 2 − 1)/64 > (14 × 2 − 1)/196). This experiment shows that treating the dynamic demand matrix as an image, and applying standard CNN filters does not capture enough spatial correlations among related OD flows without considering their topological connections from the perspective of graphs. For more details, please refer to Batch normalization is used in the VACN component. The batch size in our experiment was set to 10, corresponding to 10 randomly sampled timestamps and all the 50 2 OD flows in each snapshot. The initial liearning rate is set to be 10 −4 with a decay rate 10 −6. We use early stopping for all the deep learning-based methods where the training process is terminated when the RMSE over validation set has not been improved for 10 successive epochs. The maximal number of epochs allowed is 100. In this section, we want to explore how some important hyperparameters of input OD flow data, for example p 1 and p 2, may affect the performance of our LSTOD model. Figure 6 (b) compares RMSE on testing data by STOD model with different data settings. Varied combinations of the short-term sequence length p 1 and the long-term day-level sequence length p 2 are studied. We can see that the best performance is achieved as (p 1, p 2) = with RMSE = 2.41. Specifically, settings with different p 1's under p 2 = 5 consistently outperform those under p 2 = 7. It may demonstrate that the shift can usually be captured within a short time range, while a longer time sequence may smooth the significance. Table 4 provides the detailed prediction for each data setting.

We propose a purely convolutional CNN model with attention mechanism to predict spatial-temporal origin-destination flows.

Represent the short summary for finding the corresponding scientific article (long)

Combining graph neural networks and the RNN graph generative model, we propose a novel architecture that is able to learn from a sequence of evolving graphs and predict the graph topology evolution for the future timesteps