Question: Represent the scientific article to find a short TLDR

Question: Deep reinforcement learning has achieved many recent successes, but our understanding of its strengths and limitations is hampered by the lack of rich environments in which we can fully characterize optimal behavior, and correspondingly diagnose individual actions against such a characterization. Here we consider a family of combinatorial games, arising from work of Erdos, Selfridge, and Spencer, and we propose their use as environments for evaluating and comparing different approaches to reinforcement learning. These games have a number of appealing features: they are challenging for current learning approaches, but they form (i) a low-dimensional, simply parametrized environment where (ii) there is a linear closed form solution for optimal behavior from any state, and (iii) the difficulty of the game can be tuned by changing environment parameters in an interpretable way. We use these Erdos-Selfridge-Spencer games not only to compare different algorithms, but also to compare approaches based on supervised and reinforcement learning, to analyze the power of multi-agent approaches in improving performance, and to evaluate generalization to environments outside the training set. Deep reinforcement learning has seen many remarkable successes over the past few years BID5 BID9. But developing learning algorithms that are robust across tasks and policy representations remains a challenge. Standard benchmarks like MuJoCo and Atari provide rich settings for experimentation, but the specifics of the underlying environments differ from each other in many different ways, and hence determining the principles underlying any particular form of sub-optimal behavior is difficult. Optimal behavior in these environments is generally complex and not fully characterized, so algorithmic success is generally associated with high scores, making it hard to analyze where errors are occurring in any sort of fine-grained sense. An ideal setting for studying the strengths and limitations of reinforcement learning algorithms would be (i) a simply parametrized family of environments where (ii) optimal behavior can be completely characterized, (iii) the inherent difficulty of computing optimal behavior is tightly controlled by the underlying parameters, and (iv) at least some portions of the parameter space produce environments that are hard for current algorithms. To produce such a family of environments, we look in a novel direction -to a set of two-player combinatorial games with their roots in work of Erdos and Selfridge BID3, and placed on a general footing by BID10. Roughly speaking, these Erdos-Selfridge-Spencer (ESS) games are games in which two players take turns selecting objects from some combinatorial structure, with the feature that optimal strategies can be defined by potential functions derived from conditional expectations over random future play. These ESS games thus provide an opportunity to capture the general desiderata noted above, with a clean characterization of optimal behavior and a set of instances that range from easy to very hard as we sweep over a simple set of tunable parameters. We focus in particular on one of the best-known games in this genre, Spencer's attacker-defender game (also known as the "tenure game"; BID10, in which -roughly speaking -an attacker advances a set of pieces up the levels of a board, while a defender destroys subsets of these pieces to try prevent any of them from reaching the final level ( FIG0). An instance of the game can be parametrized by two key quantities. The first is the number of levels K, which determines both the size of the state space and the approximate length of the game; the latter is directly related to the sparsity of win/loss signals as rewards. The second quantity is a potential function φ, whose magnitude characterizes whether the instance favors the defender or attacker, and how much "margin of error" there is in optimal play. The environment therefore allows us to study learning by the defender, or by the attacker, or in a multi-agent formulation where the defender and attacker are learning concurrently. Because we have a move-by-move characterization of optimal play, we can go beyond simple measures of reward based purely on win/loss outcomes and use supervised learning techniques to pinpoint the exact location of the errors in a trajectory of play. In the process, we are able to develop insights about the robustness of solutions to changes in the environment. These types of analyses have been long-standing goals, but they have generally been approached much more abstractly, given the difficulty in characterizing step-by-step optimally in non-trivial environments such as this one. The main contributions of this work are thus the following:1. The development of these combinatorial games as environments for studying the behavior of reinforcement learning algorithms, with sensitive control over the difficulty of individual instances using a small set of natural parameters.2. A comparison of the performance of an agent trained using deep RL to the performance of an agent trained using supervised learning on move-by-move decisions. Exploiting the fact that we can characterize optimal play at the level of individual moves, we find an intriguing phenomenon: while the supervised learning agent is, not surprisingly, more accurate on individual move decisions than the deep RL agent, the deep RL agent is better at playing the game! We further interpret this by studying fatal mistakes.3. An investigation of the way in which the success of one of the two players (defender or attacker) in training turns out to depend crucially on the algorithm being used to implement the other player. We explore properties of this other player's algorithm, and also properties of mulitagent learning, that lead to more robust policies with better generalization. This is a largely empirical paper, building on a theoretically grounded environment derived from a combinatorial game. We present learning and generalization experiments for a variety of commonly used model architectures and learning algorithms. We aim to show that despite the simple structure of the game, it provides both significant challenges for standard reinforcement learning approaches and a number of tools for precisely understanding those challenges. We first introduce the family of Attacker-Defender Games BID10, a set of games with two properties that yield a particularly attractive testbed for deep reinforcement learning: the ability to continuously vary the difficulty of the environment through two parameters, and the existence of a closed form solution that is expressible as a linear model. An Attacker-Defender game involves two players: an attacker who moves pieces, and a defender who destroys pieces. An instance of the game has a set of levels numbered from 0 to K, and N pieces that are initialized across these levels. The attacker's goal is to get at least one of their pieces to level K, and the defender's is to destroy all N pieces before this can happen. In each turn, the attacker proposes a partition A, B of the pieces still in play. The defender then chooses one of the sets to destroy and remove from play. All pieces in the other set are moved up a level. The game ends when either one or more pieces reach level K, or when all pieces are destroyed. FIG0 shows one turn of play. With this setup, varying the number of levels K or the number of pieces N changes the difficulty for the attacker or the defender. One of the most striking aspects of the Attacker-Defender game is that it is possible to make this tradeoff precise, and en route to doing so, also identify a linear optimal policy. We start with a simple special case -rather than initializing the board with pieces placed arbitrarily, we require the pieces to all start at level 0. In this special case, we can directly think of the game's difficulty in terms of the number of levels K and the number of pieces N. Theorem 1. Consider an instance of the Attacker-Defender game with K levels and N pieces, with all N pieces starting at level 0. Then if N < 2 K, the defender can always win. There is a simple proof of this fact: the defender simply always destroys the larger one of the sets A or B. In this way, the number of pieces is reduced by at least a factor of two in each step; since a piece must travel K steps in order to reach level K, and N < 2 K, no piece will reach level K.When we move to the more general case in which the board is initialized at the start of the game with pieces placed at arbitrary levels, it will be less immediately clear how to define the "larger" one of the sets A or B. We therefore give a second proof of Theorem 1 that will be useful in these more general settings. This second proof BID10 ) uses Erdos's probabilistic method and proceeds as follows: for any attacker strategy, assume the defender plays randomly. Let T be a random variable for the number of pieces that reach level K. Then T = T i where T i is the indicator that piece i reaches level K. DISPLAYFORM0 as the defender is playing randomly, any piece has probability 1/2 of advancing a level and 1/2 of being destroyed. As all the pieces start at level 0, they must advance K levels to reach the top, which happens with probability 2 −K. But now, by choice of N, we have that i 2 −K = N 2 −K < 1. Since T is an integer random variable, E [T] < 1 implies that the distribution of T has nonzero mass at 0 -in other words there is some set of choices for the defender that guarantees destroying all pieces. This means that the attacker does not have a strategy that wins with probability 1 against random play by the defender; since the game has the property that one player or the other must be able to force a win, it follows that the defender can force a win. Now consider the general form of the game, in which the initial configuration can have pieces at arbitrary levels. Thus, at any point in time, the state of the game can be described by a K-dimensional vector S = (n 0, n 1, ..., n K), with n i the number of pieces at level i. Extending the argument used in the second proof above, we note that a piece at level l has a 2 DISPLAYFORM1 chance of survival under random play. This motivates the following potential function on states: Definition 1. Potential Function: Given a game state S = (n 0, n 1, ..., n K), we define the potential of the state as DISPLAYFORM2 Note that this is a linear function on the input state, expressible as φ(S) = w T · S for w a vector with w l = 2 −(K−l). We can now state the following generalization of Theorem 1.Theorem 2. Consider an instance of the Attacker-Defender game that has K levels and N pieces, with pieces placed anywhere on the board, and let the initial state be S 0. Then (a) If φ(S 0) < 1, the defender can always win (b) If φ(S 0) ≥ 1, the attacker can always win. One way to prove part (a) of this theorem is by directly extending the proof of Theorem 1, with This definition of the potential function gives a natural, concrete strategy for the defender: the defender simply destroys whichever of A or B has higher potential. We claim that if φ(S 0) < 1, then this strategy guarantees that any subsequent state S will also have φ(S) < 1. Indeed, suppose (renaming the sets if necessary) that A has a potential at least as high as B's, and that A is the set destroyed by the defender. Since φ(B) ≤ φ(A) and φ(A) + φ(B) = φ(S) < 1, the next state has potential 2φ(B) (double the potential of B as all pieces move up a level) which is also less than 1. In order to win, the attacker would need to place a piece on level K, which would produce a set of potential at least 1. Since all sets under the defender's strategy have potential strictly less than 1, it follows that no piece ever reaches level K. DISPLAYFORM3 If φ(S 0) ≥ 1, we can devise a similar optimal strategy for the attacker. The attacker picks two sets A, B such that each has potential ≥ 1/2. The fact that this can be done is shown in Theorem 3, and in BID10. Then regardless of which of A, B is destroyed, the other, whose pieces all move up a level, doubles its potential, and thus all subsequent states S maintain φ(S) ≥ 1, ing in an eventual win for the attacker. The Atari benchmark BID5 is a well known set of tasks, ranging from easy to solve (Breakout, Pong) to very difficult (Montezuma's Revenge). BID2 proposed a set of continuous environments, implemented in the MuJoCo simulator BID13. An advantage of physics based environments is that they can be varied continuously by changing physics parameters BID7, or by randomizing rendering BID12. Deepmind Lab BID0 ) is a set of 3D navigation based environments. OpenAI Gym BID1 contains both the Atari and MuJoCo benchmarks, as well as classic control environments like Cartpole BID11 and algorithmic tasks like copying an input sequence. The difficulty of algorithmic tasks can be easily increased by increasing the length of the input. Our proposed benchmark merges properties of both the algorithmic tasks and physics-based tasks, letting us increase difficulty by discrete changes in length or continuous changes in potential. From Section 2, we see that the Attacker-Defender games are a family of environments with a difficulty knob that can be continuously adjusted through the start state potential φ(S 0) and the number of levels K. In this section, we describe a set of baseline on Attacker-Defender games that motivate the exploration in the remainder of this paper. We set up the Attacker-Defender environment as follows: the game state is represented by a K + 1 dimensional vector for levels 0 to K, with coordinate l representing the number of pieces at level l. For the defender agent, the input is the concatenation of the partition A, B, giving a 2(K + 1) dimensional vector. The game start state S 0 is initialized randomly from a distribution over start states of a certain potential. We first look at training a defender agent against an attacker that randomly chooses between (mostly) playing optimally, and (occasionally) playing suboptimally, with the Disjoint Support Strategy. This strategy unevenly partitions the occupied levels between A, B so that one set has higher potential than the other, with the proportional difference between the two sets being sampled randomly. Note that this strategy gives rise to very different states A, B (uneven potential, disjoint occupied levels) than the optimal strategy, and we find that the model learns a much more generalizable policy when mixing between the two (Section 6).When testing out reinforcement learning, we have two choices of difficulty parameters. The potential of the start state, φ(S 0), changes how optimally the defender has to play, with values close to 1 giving much less leeway for mistakes in valuing the two sets. Changing K, the number of levels, directly affects the sparsity of the reward, with higher K ing in longer games and less feedback. Additionally, K also greatly increases the number of possible states and game trajectories (see Theorem 4). theoretically expressive enough to learn the optimal policy for the defender agent. In practice, we see that for many difficulty settings and algorithms, RL struggles to learn the optimal policy and performs more poorly than when using deeper models (compare to Figure 3). An exception to this is DQN which performs relatively well on all difficulty settings. Recall that the optimal policy can be expressed as a linear network, with the weights given by the potential function, Definition 1. We therefore first try training a linear model for the defender agent. We evaluate Proximal Policy Optimization (PPO), Advantage Actor Critic (A2C) BID6, and Deep Q-Networks (DQN) BID5, using the OpenAI Baselines implementations BID4. Both PPO and A2C find it challenging to learn the harder difficulty settings of the game, and perform better with deeper networks FIG2 ). DQN performs surprisingly well, but we see some improvement in performance variance with a deeper model. In summary, while the policy can theoretically be expressed with a linear model, empirically we see gains in performance and a reduction in variance when using deeper networks (c.f. Figures 3, 4.) Having evaluated the performance of linear models, we try a deeper model for our policy net: a fully connected neural network with two hidden layers of width 300. (Hyperparameters were chosen without extensive tuning and by trying a few different possible settings. We found that two hidden layers generally performed best and the width of the network did not have much effect on the resutls.) Identically to above, we evaluate PPO, A2C and DQN on varying start state potentials and K. Each algorithm is run with 3 random seeds, and in all plots we show minimum, mean, and maximum performance. of potential (top and bottom row) with a deep network. All three algorithms show a noticeable variation in performance over different difficulty settings, though we note that PPO seems to be more robust to larger K (which corresponds to longer episodes). A2C tends to fare worse than both PPO and DQN.. Unsurprisingly, we see that supervised learning is better on average at getting the ground truth correct move. However, RL is better at playing the game: a policy trained through RL significantly outperforms a policy trained through supervised learning (right pane), with the difference growing for larger K. One remarkable aspect of the Attacker-Defender game is that not only do we have an easily expressible optimal policy, but we know the ground truth on a per move basis. We can thus compare RL to a Supervised Learning setup, where we classify the correct action on a large set of sampled states. To carry out this test in practice, we first train a defender policy with reinforcement learning, saving all observations seen to a dataset. We then train a supervised network (with the same architecture as the defender policy) to classify the optimal action. This ensures both methods see exactly the same data points. We then test the supervised network on how well it can play. The , shown in FIG4 are counter intuitive. Supervised learning (unsurprisingly) has a higher proportion of correct moves: keeping count of the ground truth correct move for each turn in the game, the trained supervised policy network has a higher proportion of ground truth correct moves in play. However, despite this, reinforcement learning is better at playing the game, winning a larger proportion of games. These are shown in FIG4 for varying choices of K.We conjecture that reinforcement learning is learning to focus most on moves that matter for winning. To investigate this conjecture, we perform two further experiments. Define a fatal mistake to be when the defender moves from a winning state (potential < 1) to a losing state (potential > 1) due to an incorrect move. We count the number of fatal mistakes made by the trained supervised policy, and trained RL policy. The are shown in the left pane of FIG5. We see that supervised learning is much more prone to make fatal mistakes, with a sharp increase in fatal mistakes for larger K, supporting its sharp decrease in performance. We also look at where mistakes are made by RL and Supervised Learning based on distance of the move from the end of the game. We find that RL is better at the final couple of moves, and then consistently better in most of the earlier parts of the game. This contrast forms an interesting counterpart to recent findings of BID9, who in the context of Go also compared reinforcement learning to supervised approaches. A key distinction is that their supervised work was relative to a heuristic objective, whereas in our domain we are able to compare to provably optimal play. Returning to our RL Defender Agent, we would like to know how robust its learned policy is. In particular, as we have so far been training our agent with a randomized but hard coded attacker, we would like to test how sensitive a defender agent is to the particular attacker strategy. We investigate this in FIG7 where we first train a defender agent on the optimal attacker and test on the disjoint support attacker. We notice a large drop in performance when switching from the optimal attacker to (less than 1.0 potential) to losing state (greater than 1.0 potential)) made by supervised learning compared to RL. We find that Supervised Learning makes many more fatal mistakes, explaining its collapse in performance. (right pane) plot showing when (measured as distance to end game) RL and supervised learning make mistakes. RL is more accurate than supervised learning at predicting the right action for the final couple of moves, and then drops quickly to a constant, whereas supervised learning is less accurate right at the very end and drops more slowly but much further, having lower accuracy than RL for many of the earlier moves. and then tested on (a) another optimal attacker environment (b) the disjoint support attacker environment. The left pane shows the ing performance drop when switching to testing on the same opponent strategy as in training to a different opponent strategy. The right pane shows the of testing on an optimal attacker vs a disjoint support attacker during training. We see that performance on the disjoint support attacker converges to a significantly lower level than the optimal attacker. Figure 8: Performance of PPO and A2C on training the attacker agent for different difficulty settings. DQN performance was very poor (reward < −0.8 at K = 5 with best hyperparams). We see much greater variation of performance with changing K, which now affects the sparseness of the reward as well as the size of the action space. There is less variation with potential, but we see a very high performance variance (top right pane) with lower (harder) potentials. the disjoint support attacker. As we know there exists an optimal policy which generalizes perfectly across all attacker strategies, this suggests that the defender is overfitting to the particular attacker strategy. One way to mitigate this overfitting issue is to set up a method of also training the attacker, with the goal of training the defender against a learned attacker, or even better, in the multiagent setting. However, determining the correct setup to train the attacker agent first requires devising a tractable parametrization of the action space. A naive implementation of the attacker would be to have the policy output how many pieces should be allocated to A for each of the K + 1 levels (as described in BID10). This can grow exponentially in K, which is clearly impractical. To address this, we first prove a theorem that enables us to show that we can parametrize an optimal attacker with a much smaller action space. Theorem 3. For any Attacker-Defender game with K levels, start state S 0 and φ(S 0) ≥ 1, there exists a partition A, B such that φ(A) ≥ 0.5, φ(B) ≥ 0.5, and for some l, A contains pieces of level i > l, and B contains all pieces of level i < l. Proof. For each l ∈ {0, 1, . . ., K}, let A l be the set of all pieces from levels K down to and excluding level l, with A K = ∅. We have φ(A i+1) ≤ φ(A i), φ(A K) = 0 and φ(A 0) = φ(S 0) ≥ 1. Thus, there exists an l such that φ(A l) < 0.5 and φ(A l−1) ≥ 0.5. If φ(A l−1) = 0.5, we set A l−1 = A and B the complement, and are done. So assume φ(A l) < 0.5 and φ(A l−1) > 0.5Since A l−1 only contains pieces from levels K to l, potentials φ(A l) and φ(A l−1) are both integer multiples of 2 −(K−l), the value of a piece in level l. Letting φ(A l) = n · 2 −(K−l) and φ(A l−1) = m · 2 −(K−l), we are guaranteed that level l has m − n pieces, and that we can move k < m − n pieces from A l−1 to A l such that the potential of the new set equals 0.5. Figure 9: Performance of attacker and defender agents when learning in a multiagent setting. In the top panes, solid lines denote attacker performance. In the bottom panes, solid lines are defender performance. The sharp changes in performance correspond to the times we switch which agent is training. We note that the defender performs much better in the multiagent setting: comparing the top and bottom left panes, we see far more variance and lower performance of the attacker compared to the defender performance below. Furthermore, the attacker loses to the defender for potential 1.1 at K = 15, despite winning against the optimal defender in Figure 8. We also see (right panes) that the attacker has higher variance and sharper changes in its performance even under conditions when it is guaranteed to win. This theorem gives a different attacker parametrization. The attacker outputs a level l. The environment assigns all pieces before level l to A, all pieces after level l to B, and splits level l among A and B to keep the potentials of A and B as close as possible. Theorem 3 guarantees the optimal policy is representable, and the action space linear in K instead of exponential in K.With this setup, we train an attacker agent against the optimal defender with PPO, A2C, and DQN. The DQN were very poor, and so we show for just PPO and A2C. In both algorithms we found there was a large variation in performance when changing K, which now affects both reward sparsity and action space size. We observe less outright performance variability with changes in potential for small K but see an increase in the variance (Figure 8). With this attacker training, we can now look at learning in a multiagent setting. We first explore the effects of varying the potential and K as shown in Figure 9. Overall, we find that the attacker fares worse in multiagent play than in the single agent setting. In particular, note that in the top left pane of Figure 9, we see that the attacker loses to the defender even with φ(S 0) = 1.1 for K = 15. We can compare this to Figure 8 where with PPO, we see that with K = 15, and potential 1.1, the single agent attacker succeeds in winning against the optimal defender. defender. The figure single agent defender trained on the optimal attacker and then tested on the disjoint support attacker and a multiagent defender also tested on the disjoint support attacker for different values of K. We see that multiagent defender generalizes better to this unseen strategy than the single agent defender. Finally, we return again to our defender agent, and test generalization between the single and multiagent settings. We train a defender agent in the single agent setting against the optimal attacker, and test on a an attacker that only uses the Disjoint Support strategy. We also test a defender trained in the multiagent setting (which has never seen any hardcoded strategy of this form) on the Disjoint Support attacker. The are shown in FIG0. We find that the defender trained as part of a multiagent setting generalizes noticeably better than the single agent defender. We show the over 8 random seeds and plot the mean (solid line) and shade in the standard deviation. In this paper, we have proposed Erdos-Selfridge-Spencer games as rich environments for investigating reinforcement learning, exhibiting continuously tunable difficulty and an exact combinatorial characterization of optimal behavior. We have demonstrated that algorithms can exhibit wide variation in performance as we tune the game's difficulty, and we use the characterization of optimal behavior to expose intriguing contrasts between performance in supervised learning and reinforcement learning approaches. Having reformulated the to enable a trainable attacker, we have also been able to explore insights on overfitting, generalization, and multiagent learning. We also develop further in the Appendix, including an analysis of catastrophic forgetting, generalization across different values of the game's parameters, and a method for investigating measures of the model's confidence. We believe that this family of combinatorial games can be used as a rich environment for gaining further insights into deep reinforcement learning. On the left we train on different potentials and test on potential 0.99. We find that training on harder games leads to better performance, with the agent trained on the easiest potential generalizing worst and the agent trained on a harder potential generalizing best. This is consistent across different choices of test potentials. The right pane shows the effect of training on a larger K and testing on smaller K. We see that performance appears to be inversely proportional to the difference between the train K and test K. In the main text we examined how our RL defender agent performance varies as we change the difficulty settings of the game, either the potential or K. Returning again to the fact that the AttackerDefender game has an expressible optimal that generalizes across all difficulty settings, we might wonder how training on one difficulty setting and testing on a different setting perform. Testing on different potentials in this way is straightforwards, but testing on different K requires a slight reformulation. our input size to the defender neural network policy is 2(K + 1), and so naively changing to a different number of levels will not work. Furthermore, training on a smaller K and testing on larger K is not a fair test -the model cannot be expected to learn how to weight the lower levels. However, testing the converse (training on larger K and testing on smaller K) is both easily implementable and offers a legitimate test of generalization. We find (a subset of plots shown in FIG0) that when varying potential, training on harder games in better generalization. When testing on a smaller K than the one used in training, performance is inverse to the difference between train K and test K. Recently, several papers have identified the issue of catastrophic forgetting in Deep Reinforcement Learning, where switching between different tasks in destructive interference and lower performance instead of positive transfer. We witness effects of this form in the Attacker-Defender games. As in Section 7, our two environments differ in the K that we use -we first try training on a small K, and then train on larger K. For lower difficulty (potential) settings, we see that this curriculum learning improves play, but for higher potential settings, the learning interferes catastrophically, FIG0 The significant performance drop we see in FIG7 motivates investigating whether there are simple rules of thumb that the model has successfully learned. Perhaps the simplest rule of thumb is learning the value of the null set: if one of A, B (say A) consists of only zeros and the other (B) has some pieces, the defender agent should reliably choose to destroy B. Surprisingly, even this simple rule of thumb is violated, and even more frequently for larger K, FIG0. We can also test to see if the model outputs are well calibrated to the potential values: is the model more confident in cases where there is a large discrepancy between potential values, and fifty-fifty where the potential is evenly split? The are shown in FIG0. In the main paper, we mixed between different start state distributions to ensure a wide variety of states seen. This begets the natural question of how well we can generalize across start state distribution if we train on purely one distribution. The in FIG0 show that training naively Confidence as a function of potential difference between states. The top figure shows true potential differences and model confidences; green dots are moves where the model prefers to make the right prediction, while red moves are moves where it prefers to make the wrong prediction. The right shows the same data, plotting the absolute potential difference and absolute model confidence in its preferred move. Remarkably, an increase in the potential difference associated with an increase in model confidence over a wide range, even when the model is wrong. In fact, the amount of possible starting states for a given K and potential φ(S 0) = 1 grows super exponentially in the number of levels K. We can state the following theorem:Theorem 4. The number of states with potential 1 for a game with K levels grows like 2 DISPLAYFORM0 We give a sketch proof. Proof. Let such a state be denoted S. Then a trivial upper bound can be computed by noting that each s i can take a value up to 2 (K−i), and producting all of these together gives roughly 2 K/2.For the lower bound, we assume for convenience that K is a power of 2 (this assumption can be avoided). Then look at the set of non-negative integer solutions of the system of simultaneous equations a j−1 2 1−j + a j 2 −j = 1/K where j ranges over all even numbers between log(K) + 1 and K. The equations don't share any variables, so the solution set is just a product set, and the number of solutions is just the product As the optimal defender policy is expressible as a linear model, we empirically investigate whether depth is helpful. We find that even with a temperature included, linear models perform worse than models with one or two hidden layers.

We adapt a family of combinatorial games with tunable difficulty and an optimal policy expressible as linear network, developing it as a rich environment for reinforcement learning, showing contrasts in performance with supervised learning, and analyzing multiagent learning and generalization.

represent the tldr for finding the corresponding scientific article

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Adoption of deep learning in safety-critical systems raise the need for understanding what deep neural networks do not understand. Several methodologies to estimate model uncertainty have been proposed, but these methodologies constrain either how the neural network is trained or constructed. We present Outlier Detection In Neural networks (ODIN), an assumption-free method for detecting outlier observations during prediction, based on principles widely used in manufacturing process monitoring. By using a linear approximation of the hidden layer manifold, we add prediction-time outlier detection to models after training without altering architecture or training. We demonstrate that ODIN efficiently detect outliers during prediction on Fashion-MNIST, ImageNet-synsets and speech command recognition. Thanks to the powerful transformations learned by deep neural networks, deep learning is applied in an increasing number of applications. But deep neural networks, as all data-driven models, tend to fail when input data differs from training data. Adopting deep learning in increasingly complex and possibly safety-critical systems makes it crucial to know not only whether the model's predictions are accurate, but also whether the model should predict at all. If the model is able to detect outlier observations post-training directly, the system can fall back to safe behaviour minimizing negative consequences of faulty predictions. By understanding the limits of models' learned representations and detecting when observations are not recognized, autonomous decision making based on deep learning can be improved. In manufacturing process control, predictive models have long been used to predict process outcomes as well as detecting outlier input for decades BID8 BID17 BID9. A widely used model for this purpose is Partial Least Squares regression (PLS) BID33, which project input data onto a set of linear latent variables prior to prediction. In the latent variable space, the distance from new observations to the training data distribution is used to detect outlier observations. The latent variables can also be used to approximate input observations, meaning that outliers can also be detected by measuring the distance to the latent variable subspace itself. However, the layers of a neural network learn a non-linear mapping from input to output data spaces rather than a single linear subspace. This makes it difficult to directly determine the limits of a neural network model's knowledge in the same manner as for a PLS model. In this paper we present Outlier Detection In Neural networks (ODIN), a method for detecting outliers during prediction in deep neural networks. Based on principles long used in manufacturing process control, we propose using a linear approximation of intermediate activations to provide a fixed representation of the training data. By comparing new data to this fixed representation we are able to detect outliers during prediction without imposing constraints on architecture or training. This allows us to use ODIN as a plug-in method allowing reliable and safe autonomous decision making based on deep learning. A wide collection of methods allowing neural networks to describe uncertainty in predictions, which can be used to determine if new observations are outliers, have been proposed. For decades, many methods have been formulated within a Bayesian framework BID7 BID21 allowing neural networks to predict probability distributions rather than point inferences. The predictive uncertainty can then be estimated by the entropy or variance of the predicted distribution. BID10 proposed MC-dropout, using prediction time dropout and Monte-Carlo sampling. In summary, MC-dropout make multiple predictions per inference while the network is randomly perturbed by drop-out which in a predicted distribution. A number of alternatives to using dropout to perturb Monte-Carlo samples have been proposed in recent years including: sampling based on batch-normalization parameters BID29, model ensembles BID18, multiple prediction heads in a shared base network BID25 BID14 BID22, variational inference of weight distribution instead of regular point weights BID0 and Laplace approximation of distributions from existing weights BID26. However, the mentioned methods constrain either how the network is constructed BID18 BID25 BID14 BID22 or how the network is trained BID10 BID29 limiting their use in systems already in production. Several methods also rely on multiple inferences per prediction BID10 BID29 BID0 BID26. This limits their use in real-time systems or systems with limited computational resources. An alternative approach for estimating uncertainty in classification problems is presented by where linear classifiers are trained to classify the target output given intermediate layers of a given base model. The linear classifier outputs are then fed to a meta-model that is trained to estimate whether or not the base model is correct. Another alternative approach is proposed by BID19 that leverage Generative Adversarial Networks (GANs) BID12 to augment the original dataset with border-line outliers. A deep neural classifier is then trained to output high uncertainty for outlier observations and low uncertainty for the original observations. This method does however involve training a GAN that can be difficult to train to convergence BID23.Anomaly detection is closely related to prediction time outlier detection, and there are many methods for flagging deviating observations. Non neural-methods include one-class support vector machines BID27, local observation density BID2, distances BID16, isolation forests BID20 and many others. A multitide of methods based on deep neural networks, typically autoencoders have been developed as well BID35 BID3 BID36 BID24. Of particular relevance to this work is BID24, that use reconstruction residual as metric to flag outliers. Important to note is that outlier detection systems are based on training a separate model to detect deviant observations. Prediction time outlier detection, on the other hand, describes the limits of a predictive model's knowledge. In this section we briefly describe the Partial Least Squares regression model, and how its latent variable approximation of the input data space is used to detect outliers after training. We then describe how we can apply similar principles in neural networks by using a linear approximation of the hidden layer manifold, in a method we call Outlier Detection In Neural networks (ODIN). Partial least squares regression (PLS) BID32 BID11 ) is a widely used regression model within manufacturing process control. Similar to Principal Component Analysis (PCA), PLS assumes that high-dimensional data resides in a sub-space of the original data space spanned by so called latent variables and formulated in terms of matrix decomposition. The PLS model is summarized as: DISPLAYFORM0 where the n × m input matrix X is decomposed into n × k latent variable matrix T = [t 1 ... t k] and m × k loading matrix P = [p 1 ... p k] with residual matrix E. The n × p response matrix Y is predicted using T multiplied with response weight matrix C = [c 1 ... c k] with residuals F. The latent variable-matrix T spans a orthogonal subspace of the data-matrix X and maximize the covariance between X and Y. Note that the PLS model of X is similar to how PCA approximates the data through matrix decomposition but PCA finds latent variables t i that maximize the variance in X rather than the covariance between two matrices. The columns in T are typically calculated sequentially, where the first column t 1 is found through basis-vectors w 1 ∈ R m and c 1 ∈ R p solving the optimization problem: DISPLAYFORM1 The corresponding loading vector p 1 is then chosen so thatX 1 = X − t 1 p T 1 is uncorrelated with t 1. This is achieved by selecting p 1 as: DISPLAYFORM2 The subsequent vectors t i, w i, p i where i ∈ [2, ..., k] are then calculated by repeating equations 2 and 3 usingX DISPLAYFORM3 The latent variable formulation means that PLS carries its own model of the training data and provides two ways of detecting outliers during prediction. Since new observations are projected to the low-dimensional sub-space spanned by the latent variables, both distance to the sub-space itself and distance to training observations within the sub-space can be used to detect outliers. Distance to sub-space is typically measured using the residual sum of squares (RSS). RSS for a new observation row vector x new ∈ R p is given by: DISPLAYFORM4 where x new is approximated as DISPLAYFORM5 There are several ways to estimate the distance to training observations within the sub-space and a common choice is the Mahalanobis distance. The Mahalanobis distance is a well-used statistical distance measuring how many standard deviations away an observations is from the origin in a multivariate probability normal distribution. Given a fitted PLS model, the training data projections can be approximated as a multivariate normal distribution with covariance matrix C T = E(T T T). Then the Mahalanobis distance for x new is given by: DISPLAYFORM6 Alternatively, to compensate for using a linear model of a possibly non-linear manifold, a density based metric within the latent variable space may be used. For instance, by using the Local Outlier Factor (LOF) BID2, observations within low-density regions may be flagged as outliers instead of only using the Mahalanobis distance. In contrast to PLS, data are not typically linearly mapped to a single sub-space in deep neural networks. Instead, a neural network performs as a nested series of non-linear transformations. That is, the activation vector a i of an observation vector x from a layer i is given by: DISPLAYFORM0 with weight-matrices W k and activation functions f k.According to the manifold hypothesis, data is assumed to reside near a region of low dimensionality that may be highly entangled. One possible explanation for why deep neural networks work well is that they are able to disentangle complicated manifolds BID1. If deep neural networks disentangle manifolds, we may find a transformation from a complicated data manifold to a manifold that is approximately linear. If this hypothesis holds true, we can apply a simple trick to add prediction time outlier detection to neural network models by building a model of the data representation within the model. Given n × m activation matrix DISPLAYFORM1 T of training data X, where the row vectors a i,k are given by equation 6, we approximate the activation manifold using PCA as: DISPLAYFORM2 where the n × k latent variable matrix T Ai contain the projections of the A i onto the orthonormal sub-space spanned by columns of the m × k loading matrix P Ai. Now we have a fixed orthogonal approximation of the activation manifold that we can use to detect outliers during prediction analogously to PLS (see 3.1). Meaning that we can measure the distance from new observations to the activation manifold as the residual sum of squares similar to equation 4 using the observation activation a i,new with projection t ai,new = a i,new P Ai: DISPLAYFORM3 Similarily, the distance from training observations within the manifold can be measured using Mahalanobis distance or density based approaches as the Local Outlier Factor within the linear approximation. For the Mahalanobis distance, the covariance matrix of the activation projections DISPLAYFORM4 We choose to call our method Outlier Detection In Neural networks, ODIN, since we use the intermediate data representations within the neural network itself. In contrast to common Bayesian approaches, we do not perturb predictions to produce prediction distributions. We simply measure the distance from new observations to the training data to determine whether they are outliers or not. In the following sections we demonstrate how to detect outliers during prediction time using ODIN on different classification tasks. We choose classification tasks for demonstration since it is straightforward to simulate outliers by excluding a subset of the classes. We also explore how to choose what layer's activations to use and rank of PCA approximation. For comparison, we also perform outlier detection using MC-Dropout BID10 since it is well-established and straightforward to implement even though it has received criticism BID25. To provide a simple classification problem with outliers encountered during prediction, we use the Fashion-MNIST BID34 dataset. Fashion-MNIST consists of 70 000 greyscale 28x28 pixel images, out of which 10 000 are test set images, of ten categories of fashion products. We excluded five classes to use as outliers, including all shoes (sandals, ankle boots and sneakers) and two clothing classes (pullover and shirts). The intuition is that shoe-images are strong outliers since all shoe-related information is absent from training data, and excluded clothes are more subtle outliers since the training data contain other upper body garments. We trained a small convolutional neural network (CNN, for architecture see Figure A .1) on five out of the ten classes. We used rmsprop BID30 ) optimization, categorical cross entropy loss function, batch size 128 for 10 epochs and kept 10 % of the images as validation set and achieved a test set accuracy of 97 %. To use for outlier detection, we extracted features from both max-pooling layers (without global average pooling) for all images. We evaluate ODIN using different outlier metrics (RSS, Mahalanobis distance and LOF), five levels of explained variance (R2) of the PCA model (50-99 %) and different layers of extracted features using the area under the receiver operating characteristic curve (ROC-AUC) as performance metric, (see FIG0, complete in TAB1 .1). We calculate the ROC-AUC comparing how well test set observations are separated from outlier observations. For comparison, we also used MCdropout to calculate the image-wise entropy from 50 Monte Carlo samples per image and evaluated the using ROC-AUC in the same way as ODIN.All metrics clearly separate strong outliers (shoes) from the test set images FIG0 left) with RSS being most sucessful (ROC-AUC 0.97 compared to Mahalanobis 0.94 and LOF 0.91). There is a trend that they benefit from increased PCA R2. Surprisingly MC-dropout failed to detect shoe outliers (ROC-AUC 0.495). The subtle outliers (non-shoes) are significantly more difficult to detect FIG0, and LOF is most successful doing so (best ROC-AUC 0.71 compared to RSS 0.60, Mahalanobis 0.61 and MC-Dropout 0.63).To conclude, the Fashion-MNIST experiment show that ODIN successfully detect outliers in a simple image classification problem. Strong outliers seem to be best detected by measuring distance to manifold while subtle outliers are better detected in low-density regions of the linear approximation using LOF. In order to provide a more complex example we demonstrate prediction time outlier detection using a pre-trained CNN on image synsets from ImageNet BID6 ). We train a cat vs. dog classifier on the cat-and dog-synsets from ImageNet and used the car-and horse-synsets as outliers. We used an Inception v3-network BID28 pre-trained on ImageNet, freezing all Inception module weights during training. We replaced the penultimate layer with a hidden layer of 128 ReLu units and a single sigmoid output with 50 % dropout before and after the ReLu-layer and trained for 50 epochs using the Adam optimizer BID15 and achieved a test set accuracy of 93 %.We extracted features from each inception module in the Inception-v3 network and pooled them feature-map wise using global average pooling. For each layer of features, we performed outlier detection with five levels of explained variance (R2) for the PCA model (50-99 %) and different outlier We are able to convincingly detect cars using our cats vs. dogs classifier (best ROC-AUC for RSS, 0.96, Mahalanobis distance 0.94 and LOF 0.93). Horses are also detected as outliers even though they share visual features with cats and dogs (best ROC-AUC for RSS 0.76, Mahalanobis distance 0.75 and LOF 0.69). Since we used dropout for the last fully connected layers, we also performed MC-dropout achieving similar (ROC-AUC: 0.86 for cars, 0.61 for horses). The degree of explained variance was not as influental in this experiment as in the Fashion-MNIST experiment, but both Mahalanobis distance and LOF fail to detect both cars and horses using 50 % R2. Interestingly, the performance of all metrics peak at inception module 8 where an auxilliary output was used during training on ImageNet BID28.To conclude, the experiment on cats and dogs show that ODIN reliably detect outliers using a pretrained CNN on real-world images. ODIN performs slightly better than MC-dropout but does not rely on using dropout, or any type of constraint on the training procedure. In line with the from the Fashion-MNIST experiment, higher PCA R2 produce more reliable . To show that ODIN for prediction time outlier detection works for not only CNN-based image classification, we perform a speech command recognition experiment using a LSTM-based model. We use the Speech Commands dataset BID31 ) that consists of 105 000 short utterances of 35 words recorded at 16 kHz sampling-rate. The words includes digits zero to nine, command words yes, no, up, down, left, right, on, off, stop, go, backward, forward, follow, learn and visual. The dataset also include a set of arbitrary words bed, bird, cat, dog, happy, house, marvin, sheila, tree and wow. In our experiment, we train a classification model of both digits and command words and use the arbitrary words as outliers. We transform the utterances into 64 Mel-Frecuency Cepstral Coefficients BID5, using a frame-length of 2048 samples and frame-stride of 512 samples. We train a three layer bi-directional LSTM-model with 30 % dropout after each LSTM-layer and softmax output (see architecture in Figure C .1) for 30 epochs, using the Adam-optimizer BID15 and batch-size 512 ing in test-set accuracy of 78 % for the 25 classes. The classification accuracy is lower than the 88 % accuracy of the baseline CNN:s BID31 ), but we believe it is sufficient for demonstrating prediction time outlier detection. For outlier detection, we extracted training set features from the third LSTM-layer and fitted a PCAmodel explaining 99 % of the variance and chose RSS and Mahalanobis distance limits to be the 9th deciles of the training set distances. We then extracted features from the test set and outlier classes and projected them onto the PCA-model and calculated RSS and Mahalanobis distances. Using precision, recall and F1-score we evaluated outlier detection at the 9th deciles (see FIG2, complete in TAB1 .1). We also combined RSS and Mahalanobis distance classifications using OR and AND combined classification. For comparison, we also used MC-dropout with 10 Monte Carlo samples per utterance and calculated the sample-wise Shannon entropy. We performed outlier detection using the 9th decile of training set entropies as threshold, and evaluated MC-dropout in the same manner as ODIN.Detecting outliers in the speech-commands dataset is difficult for both ODIN and MC-dropout with best word-wise F1-scores ranging from 0.25 for tree, which is phonetically similar to three, to 0.47 for house. ODIN consistently outperform MC-dropout. Additionally, since two metrics are used, we also have the opportunity to raise precision or recall by using AND-or OR-combination of classification according to the two metrics. Depending on the application, either precision or recall may be more important than the other. The Speech Command experiment shows that ODIN performs well for recurrent neural networks on a speech recognition task in addition to image classification. We also demonstrate how it can be used in practice, by selecting classification threshold and evaluating our choice using precision and recall. We also show how combinations of the different metrics available may be used to tune the precision/recall ratio. Deep neural networks are powerful transformers that have shown great success in many applications. But, in order to adopt deep learning in safety-critical applications it is crucial to understand when new observations do not match the data used during training. To imitate linear latent variable models used in manufacturing process monitoring, we use a linear approximation of the hidden layer manifolds to measure distance to and within the manifold. We compare our to MC-dropout, a well established Bayesian approach, and consistently detect outliers post-training without imposing any constraints on either architecture or training procedure. We demonstrate our method in two image classification experiments, with and without a pre-trained network, and a speech recognition example using a recurrent neural network. By defining the limits of our neural networks' knowledge, ODIN contribute to safer use of deep learning. APPENDIX A APPENDIX Input FORMULA0 Softmax ( Input (

Represent this text to find a much much longer article about it with say hundreds-thousands of words

An add-on method for deep learning to detect outliers during prediction-time

Represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: This work introduces a simple network for producing character aware word embeddings. Position agnostic and position aware character embeddings are combined to produce an embedding vector for each word. The learned word representations are shown to be very sparse and facilitate improved on language modeling tasks, despite using markedly fewer parameters, and without the need to apply dropout. A final experiment suggests that weight sharing contributes to sparsity, increases performance, and prevents overfitting. When processing text for Natural Language Processing (NLP), one important decision to make is how to represent the words for a given model or system. For many tasks tackled by deep learning such as language modeling, language understanding, and translation, the use of word embeddings has become the standard approach. BID20 BID4 BID0 This is in part due to their ability to represent complex syntactic and semantic relationships between words as spatial relationships within the embedding dimensions BID13.Embeddings are generally implemented as a lookup table for computational efficiency. However for those unfamiliar with their use it may be beneficial to formulate them as the output of the first layer in a neural network. This is true for a layer that has one-hot feature vectors as inputs, no bias, and no activation function. For a given one-hot feature vector x, the activations of such a layer can be computed by xW, which is equivalent to selecting the row W i of the weight matrix, where x i == 1. The weight matrix or embedding lookup matrix can then be optimized via typical techniques such as gradient descent, including from subsequent layers of a DNN through back propagation. BID15 For word embeddings, the basic approach is to assign a unique vector of trainable parameters to each word in a vocabulary. These vectors are referred to in this paper as token embeddings. Token embeddings present a number of limitations. For example, any out-of-vocabulary word cannot be represented except as a pre-defined < U N K > token. A corollary of this is that the number of embeddings (and therefore trainable parameters) grows linearly with the size of the vocabulary. Furthermore, characters are ignored, meaning that potentially useful morphological information is thrown out. To get around these limitations, researchers have explored building word embeddings from lower level character representations. A variety of techniques have been presented, including the use of feedforward multi layer perceptrons (MLPs) BID3, convolutional neural networks (CNNs) BID9 BID10, and recurrent neural networks (RNNs) BID12. These character level representations of words have the advantage over token embeddings of allowing an open vocabulary, usually having fewer parameters, and improving performance by making use of information available in sub-word level features. The most successful approaches for building word embeddings from characters use CNNs. BID9 However, the architecture of CNNs is designed to identify positioninvariant features, not the specific ordering of characters that make up a word's spelling. Here we ask whether such ordering is a valuable source of information. A number of convolutional features of varying size can be used to capture some ordering, for example within each feature independently. However as the vocabulary is expanded, the number convolutional features must be increased to compensate BID9. Once convolution is performed, the used of a deep highway network, as introduced by BID18, is then needed to produce the final word embedding. The current study presents a simple fully connected architecture for combining characters. In this framework, each character is represented both by position-agnostic character embeddings and position-aware character embeddings, which we call spelling embeddings. The combination of the two allows the model to learn both position invariant features and positional features. A word embedding is then constructed by combining both the character and spelling embeddings of the word, for example by summing or by averaging them together. The ing vector is then passed through a nonlinear MLP that combines the character and spelling information to produce the final word embedding. This MLP, along with the spelling and character embeddings, were trained via gradient descent as inputs to a Recurrent Neural Network (RNN) being trained for a language modeling task. Results show that including the spelling information facilitates improvement over token embeddings despite requiring far fewer parameters. Without the position information, character embeddings alone are not sufficient in this fully connected architecture. An analysis of the learned representations at the word embedding level shows much greater sparsity for spelling embeddings than for token embeddings, and demonstrates some of the negative impacts of dropout on the representations. Finally, we compare token based models with a fully connected layer of shared weights to raw token embeddings with no weight sharing. Passing the token embeddings through a layer of shared weights is shown to drastically increase representation sparsity and prevent overfitting. Given that the character and spelling weights are heavily shared among word embeddings, this is presented as possible explanation for the spelling aware model's robustness against overfitting. Many architectures have been explored for composing word embeddings from lower level features, including the use of recurrent neural networks BID12, BID11 convolutional networks BID10, BID16, BID9, character n-grams BID1, as well as combinations of word tokens with morphological features BID2, BID3.One such architecture is to enhance token based word embeddings of Chinese words by including character embeddings BID3. Multiple approaches were explored, the simplest of which was to embed characters and build a word embedding by combining a traditional token embedding with the average of the embeddings for each character in the word: DISPLAYFORM0 Where e(i) is the character enhanced embedding for word i, T is the token embedding lookup table, T i is the token embedding vector for the word, c j is the character embedding vector for the jth letter of the word, and L i is the total number of letters in the word. There are a number of drawbacks with this approach. First, character ordering is not taken into account so the token embeddings are needed to ensure uniqueness. Second, the character embeddings were not included for words that were pre-screened for ambiguous or misleading character information, which requires a manual or heuristic pre-processing step. Finally, simply averaging the character embeddings doesnt provide an opportunity to build richer non-linear combinations such as would be possible with an MLP.Convolution neural networks (CNNs) have also been used to create word embeddings from character representations. BID9 Their character aware CNN architecture was based on a previous publication by BID10, but used more convolution features to cope with larger datasets. This approach was found to give state of the art when applied to language modeling with the popular One Billion Word Benchmark, despite using far fewer parameters than a traditional token embedding model. The use of a fully connected network with explicit positional information was not reported on. The inclusion of positional information can be handled in a variety of ways. An interesting method not explored in this work is provided by BID19, who combine positional information with each symbol in the form of unlearned sin and cosine dependant functions of varying frequencies. These functions produce repeating waveforms that allow their model to capture information about relative positions. This differs from the current study which uses learned, explicit and distinct representations for each position of each character. The task in language modeling is to assign probabilities to sentences or sequences of words. That is, we want to model the probability of of the next word in a sequence conditional on the ordered sequence of all previous words. DISPLAYFORM0 This was accomplished with the use of an RNN which produces a context vector v from previous words. The RNN was implemented with Gated Recurrent Units (GRUs), which we denote here by the function g for simplicity. A fully connected layer with weights W (s) and biases b (s) was used to project the GRU's output, v, to the target vocabulary. A softmax activation was then applied to produce a valid probability distribution, q, over the vocabulary: DISPLAYFORM1 Gradients were computed from the cross entropy between the softmax layer and expected next word in the sequence. Each batch contained sequences with a fixed length, and each sequence followed from the previous batch. Gradients were time truncated to the fixed sequence length. Gradients were also clipped to a maximum global norm to prevent them from exploding. BID14 The initial states of the GRU were only reset at the beginning of each epoch. Dropout was applied to the outputs of each RNN layer in order to regularize the RNN weights of the models. BID17 For the embedding layers, two dropout configurations were compared. The first applied dropout to the final word embedding layer and the second did not. Two datasets were evaluated. The first is a relatively small dataset consisting of the novels of The Wheel of Time, by Jordan & Sanderson (1990 BID13 . It has vocabulary of 34,594 words and was split into train/test partitions with 5,007,362 and 444,576 words respectively. The second dataset is a subset of works from Project Gutenberg Canada, which is a collection of writings that are in the Canadian public domain. This larger dataset has a vocabulary of 205,027 words and was split into train/test partitions with 63,319,830 and 7,136,409 words respectively. This is about 10% the size of the popular One Billion Word Benchmark. Both datasets were pre-processed as follows. All texts were lower-cased and separated by whitespace into words. Each word was then parsed again so that any stream of consecutive alphabetical characters were considered a token, and any stream of non-alphabetical characters were considered a token. Converting the vocabulary to lowercase removes potentially valuable information, and was done only to reduce the vocabulary size. This allowed for a speed up in experimentation and hyper-parameter tuning, as well as to fit larger models on available hardware. The token embeddings consist of a V × N lookup table of trainable parameters followed by a fully connected layer with rectified linear units (ReLUs). A graphical representation is provided in figure 1. Given a lookup table T, word index i, and a fully connected layer with matrix W (t) and bias vector b (t), the embedding funtion is: DISPLAYFORM0 An additional configuration for the tokens, referred to as raw token embeddings, was investigated with the larger dataset. These were simply presented directly the the RNN, rather than passed through a fully connected layer first. Hence: DISPLAYFORM1 As shown in figure 2, the spelling embeddings are built up from the characters in the word as follows. Two lookup tables are used. One contains position agnostic character embeddings and is of size C × N c. The other contains positional character embeddings and is of size C × L × N s. Where C is the number of characters, L is the maximum word length, N c is the size of the embedding dimension for position agnostic character embeddings, and N s is the embedding dimension for spelling embeddings. To embed a word, the embeddings for the characters in the word are first pulled from each table separately and averaged. The ing vectors from these averages are then concatenated together to produce a vector of dimensionality N c + N s. This vector is then used as input to an MLP with two ReLU layers. We denote the lookup tables for the position aware and position agnostic character embeddings as U and V, respectively. Then for a word indexed by i, the vector w (i) contains the indices corresponding to the position agnostic characters of that word. Then if L (i) is the length of the word and j indexes the character position, we formulate the concatenation of the position aware and position agnostic character representations of the word as: DISPLAYFORM2 The models were also run without the position aware spelling embeddings in order to determine the value of this information for the task. All embedding methods presented have a final embedding layer with dimensionality M, in order to ensure that the language model is given an equal capacity for information about incoming words. The number of parameters required to embed the entire vocabulary was controlled in order to prevent spelling embeddings from gaining an unfair advantage over tokens. This was accomplished by limiting the number of nodes in each layer. One of the main benefits of spelling embeddings is that the number of parameters does not grow necessarily with the vocabulary size as it does with token embeddings. The number of parameters needed to embed the vocabulary using token embeddings is computed by (V × N t) + (N t × M). The dominant term is generally the vocabulary size, V, which is much larger than the embedding dimension. For spelling embeddings an upper bound is considered because not all characters must appear in all possible word positions. This is computed by: DISPLAYFORM3 where M is the size of the fully connected layer placed between the character embeddings and the final embedding layer. TAB0 shows the specific values for the number of parameters used in our experiments. The spelling embeddings do not use significantly more parameters for the larger dataset than for the smaller because they depend on the number of characters and the lengths of words rather than on the number of words. On the larger Gutenberg dataset, spelling embeddings outperform token embeddings despite using far fewer parameters to embed each word (∼ 13M vs ∼ 82M). On the smaller Wheel of Time dataset they are on par with token embeddings. Position agnostic character alone embeddings perform worse than tokens. Performance curves are plotted in figure 3. Final performance of each model is listed in table 2. The token embeddings overfit the training data on the Wheel of Time dataset. On the Gutenberg dataset, only the raw token embeddings exhibited overfitting. spelling embeddings are far more sparse than those of token embedding. Raw token embeddings exhibit the least amount of sparsity. To get a more comprehensive view of sparsity, the Gini coefficient was applied to the embeddings of the entire vocabulary for each model run on the Gutenberg dataset. BID6 The Gini coefficient was chosen as a measure of sparsity because it has been shown to be robust under a number of metrics. BID7 FIG3 shows the distribution of sparsity across the vocabulary as measured by the Gini coefficient. Raw token embeddings are the least sparse. Token embeddings passed through a fully connected layer increase dramatically in sparsity, followed by the spelling embeddings which are the most sparse. Sparsity is also affected by dropout. Whereas dropout in greater sparsity for the majority of the token embeddings, it causes a few to lose all sparsity and become completely homogeneous. Dropout also has this homogenizing effect on some of the spelling embeddings. This work shows that a simple fully connected network is able to produce character aware word embeddings that outperform traditional token embeddings. The architecture is relatively simple compared to previous approaches that use CNNs or RNNs to combine character information. This work lacks a direct comparison to these other character aware methods, which is an obvious direction for future work. Investigation into the word embeddings produced by the presented architectures reveal a number of interesting properties. Spelling embeddings are especially resistant to overfitting compared to token embeddings, and are also significantly more sparse in their activations. Furthermore, dropout is shown to have some negative impacts on the word representations, and weight sharing is presented as a better way to regularize word embeddings. Spelling embeddings exhibit the most weight sharing, because each character embedding is shared across many words in the vocabulary. This may be a contributing factor to their increased sparsity and resistance to overfitting. Additional evidence for this is provided in the comparison of raw token embeddings to those passed through a fully connected layer. Whereas raw token embeddings share none of their weights with other words in the vocabulary, token embeddings passed through a fully connected layer share all the weights in that layer across the entire vocabulary. Not only do token embeddings enjoy increased resistance to overfitting when passed though a shared weight layer, they also become drastically more sparse. Whereas dropout is a popular technique for regularization in NLP, it can have a negative impact on the word embeddings, causing some of them to gain a Gini coefficient of 0. This suggests that these particular words have completely homogeneous representations and are indistinguishable from each other. On the smaller dataset the number of shared parameters in the fully connected layer of the token embeddings is large compared to the vocabulary size. In this case, dropout is needed to prevent overfitting. On the larger dataset, the number of shared parameters is much smaller relative to the vocabulary size. In this case dropout is not needed for the token embeddings and actually hinders them. The spelling embeddings perform worse with dropout on both datasets. The architecture presented here should be compared to the state of the art character CNN obtained on the One Billion Word benchmark. BID9 ) Also, whereas a number hyper-parameters governing the number and size of the layers were tried before the ones presented in this paper were found, other techniques such as highway networks BID18 have not yet been investigated. Furthermore, extending the concept of character aware word embeddings to the output softmax layer is another open area of research that has been tried with character CNNs BID9, but not to our knowledge with a spelling network as presented in this work.

Represent the TLDR for finding the corresponding scientific article

A fully connected architecture is used to produce word embeddings from character representations, outperforms traditional embeddings and provides insight into sparsity and dropout.

LSTMs learn long-range dependencies compositionally by building them from shorter constituents over the course of training.