Question: Represent the following document

Question: This paper we present a defogger, a model that learns to predict future hidden information from partial observations. We formulate this model in the context of forward modeling and leverage spatial and sequential constraints and correlations via convolutional neural networks and long short-term memory networks, respectively. We evaluate our approach on a large dataset of human games of StarCraft: Brood War, a real-time strategy video game. Our models consistently beat strong rule-based baselines and qualitatively produce sensible future game states. We consider the problem of joint state estimation and next-state prediction in partially observable environments with complex dynamics. We take as a concrete example the problem of defogging in the real-time strategy (RTS) video game StarCraft, which we define as predicting the features of the game state that are hidden to the player. Forward modeling, the prediction of what is going to happen next, is a core enabler both for reactive control and for longer term planning. Many researchers are attempting to build and create algorithms that are able to model the future, especially in next frame video prediction and robotic planning BID9 BID0 One particular difficulty of forward modeling is to deal with the uncertainty of making a prediction with only a partial model and a partial view of the world. BID4; BID3.In RTS games such as StarCraft, players must build an economy and control agents, called units, on a 2 dimensional grid to overcome their opponents. Several inherent limitations of any real-world setting are made explicit in such RTS games. First, by the "fog of war" which only allows players to see the surroundings of their own units and are thus unable to fully access the true game state. Second, the low-level dynamics are extremely complex, because several hundreds of agents interact together. However, there is an implicit spatio-temporal structure that makes long-term reasonning depend mostly on lower-resolution abstractions that can be obtained by averaging fine-grained characteristics over time and space. This poses a challenge for both human and computer players alike and predicting hidden information is a key step towards efficient planning in such environments, In this paper, as a first step towards learning a fully-featured forward model of the environment, the task we propose is to uncover hidden information and to predict the next state from observational data. We present a comprehensive analysis of a StarCraft Defogger, which predict features of the game at different levels of granularity: global features of the game such as the buildings of the opponent, and local features such as army density averaged by regions. Starting from a map of the environment subsampled in time and space, we propose a deep architecture of stacked long short-term memory cells applied convolutionally in an encoder-decoder architecture to predict the full state at different spatial resolutions. Individual layers of convolutional LSTMs encode the dynamics of the local features, and are aggregated in subsequent layers to model lower-resolution movements and global features. Trained on a large dataset of human replays BID8, the model significantly outperforms strong rule-based baselines on several metrics. In this paper, we look at inferring the hidden information, from partial observations, either to make use of the inferred values directly, or to use them to decide what subsequent information to gather, i.e. lift our known unknowns. We do so with a forward model (see Section 4), because it allows us to leverage the spatio-temporal structure of the game. StarCraft is an imperfect information game with a rich body of research BID13. Few works attempt to deal with the fog of war: by using particle filtering to estimate the position of the opponent units BID18, or by predicting where, how and when the opponent will attack BID15. Poker is another imperfect information game, on which research started jointly with the field of game theory. BID11 yielded AIs that beat human professionals. One can draw a distant parallel between the general problem of defogging and instance segmentation in vision (when in presence of occlusions) BID14. Learning a forward model can be used to do video frame prediction BID10. Methods predicting in feature space include BID9, in which the authors predict future frames in the semantic segmentation space. Forward models have a rich history in control theory, various parametric and nonparametric methods were proposed for optimal filtering of different classes of stochastic systems BID5 ). Attempts were made to treat partial observations cases BID7, but not in the case of branching processes. BID12 enhances the exploration of a deep reinforcement algorithm with a learned predictive model in Atari. BID19 learn a forward model (jointly with a model-free controller) and use it for planning in Sokoban and MiniPacman. Finally, BID17 estimate combat outcomes in StarCraft by employing forward models in a Monte Carlo Tree Search algorithm.3 "DEFOGGING": PROBLEM STATEMENT "Defogging" consists of recovering the whole state s t from partial observations o 1...t in imperfect information games. We explain why it matters in RTS games. Let {U} n denote the set of all sets {u 1, ..., u n} consisting of n units, u i ∈ U with U being unit types. As a state space we shall use S = N n=1 {U} n where N is the maximum number of player's units. We will consider two-player games for which the full game state at time t is s t = (s DISPLAYFORM0 t) ∈ S 2. At each time-step t, each player p receives observations of state s t. We assume that each player fully observes her own state s These visibility rules for player p can be represented as a function ζ (p): S 2 → S which maps a state s t to {v ∈ {s DISPLAYFORM1 The task of defogging, akin to filtering, consists in deriving the full state s t from the observations of one player o DISPLAYFORM2 . Hence, the partial observation setting considered here is different from the classical one, where only diffusion in the state space considered or the moments of branching are supposed to be known. In Real-Time Strategy games, the task of defogging poses several challenges. First, it requires remembering virtually everything that was shown even when those units are hidden under the fog of war again (long term memory), with as many characteristics as possible, as seeing a unit of a given type at a given location at a given time may represent different information sets (or lead to different belief states) than if any of these characteristics was changed. Second, excellent performance in defogging can only be achieved by a model that can reason about this memory and apply inference steps that correspond to the game dynamics. Third, such a model can leverage the correlations of observations (due to the distribution over valid strategies), but should not overfit those. Finally, in most RTS games, inference in such a model from raw state representation is a real computational burden. As a consequence, it is necessary to formulate abstractions for making model training and evaluation tractable. Even though it is difficult to even estimate the set of valid states for s t with o 1...t precisely, humans have no difficulty in obtaining a useful estimate of the state of the opponent. It allows them to rule out large subsets of the possible states, which would reduce the computational cost of any algorithmic approach (even for model-free control, it would lower the variability in the input). Also, their estimate of the state of the opponent allows them to efficiently gather information (known unknowns). Having a good enough defogger would allow to test direct perfect information approaches in real RTS settings. Finally, human professional players exhibit theory of mind (recursive, counterfactual, reasoning about both players strategies) based on their inferred state. Algorithmic defoggers may enable a more game theoretic approach to the abstract strategic view of RTS games. We propose models based on an encoder-decoder architecture, as baselines for the defogging problem. We consider the formulated task as a parametric estimation problem. The goal is to reconstruct some coarse representation of s t from the observations o 1...t and predict both the number of units n in the opponent's state and their approximate states {u 1, . . . ., u n} that include unit type and location. We take two types of state representation in our approach: 1) unit counts per type and map region, and 2) unit type existence in a map region. The map is split into a grid of H × W cells. For each cell (c x, c y), we compute x (p) t,cx,cy ∈ N d which contains counts of units of player p for each unit type, and DISPLAYFORM0 t,cx,cy ∈ {0, 1} d that accounts for unit type presence or absence in the cell. Here d is the number of different unit types. To estimate parameters, we consider two types of loss functions: Huber ("smooth L1") loss L r (x, x) for unit type counts and binary cross-entropy loss L c (ŷ, y) for unit presence. We denoted x = (x We restricted the family of models to explore by having an encoder-decoder approach (as shown in FIG2), in which we encode x with a ConvNet (E), which does poolings in c x and c y coordinates if necessary so that we only have one vector per time-step. Then we apply a temporal inference mechanism and memory (R), for which we mostly experimented with recurrent neural networks. We tried using a simple temporal convolution, but did not notice significantly increased improvements in our metrics. We concatenate the localized embeddings of x with the output of R, and we decode with a ConvNet (D). Finally we apply a regression head (H r) and a classification head (H c) to each of the decoded vectors in each coordinate. In practice we do 2D spatial pooling before the classification head H c. DISPLAYFORM1 Because the individual statistics of each unit are difficult to predict on a macro level, and the resolution and dimensionality would be too high to be efficiently tractable, we downsample the state space into the counts of units per unit type, in evenly spaced blocks, at larger time steps than the game engine tick. The dimensions of the most popular (and representative) map are 512 × 512 walktiles, we do different models with spatially sum-pooled tiles from 32 × 32 with a striding of 32, up to 128 × 128 with a striding of 64 walktiles. Precisely, when the game map has a true height and width of H 0 and W 0, we apply a grid of size g to downsample it into a map of height H = H0 g and W = W0 g Our feature map of height H and width W is of size T × C × H × W, where T is time, C has the first half reserved for our own units and the second for enemy units. To produce coarse frame predictions, we do not need to do so at a small time difference. However, skipping a large number of frames between each prediction runs into the problem where during a period of time, a unit may appear and disappear into the fog of war, as we uncover a part but cover it up soon after. If this quick glance of the enemy is not featurized, we lose key information during the scouting process. Thus, we combine frames instead of simply skipping, and featurize units by their last seen location in a set of frames. With a step of s frames, we downsample the game in time to create T = T0 s, with the real game taking T 0 frames. The full game state holds auxiliary information that are essential to making good predictions in the future. The opponent faction (or race), if selected, is key to predicting the type of units that we need to predict. The map layout is key to predicting unit movement, as ground units cannot walk across cliffs, as well as give restrictions as to the starting locations (before any opponent units has been seen). We featurize the faction as a static embedding, replicated H × W times and concatenated to each time slice along the feature dimension We also have a ConvNet (M) that takes in features specific to each map: the walkability of each tile, buildability of each tile, ground height of each tile, and starting locations. This ConvNet is designed with the same grid size and stride as the featurizer, to ensure a H × W output size. We consider several different models, and we have done a search through model space to find the most effective models for this problem. We employ random hyperparameter search followed by manually guided random plus grid searches to find the best hyperparameters for our models. We did a preliminary search over grid sizes, and choose a grid of 32 × 32 predicting 15 seconds at a time to do most experiments on. We reason that 30 seconds is sufficient for some fast units to reach across the map, so it should be a good upper bound on how well we can expect a defogger to perform. We also present for a grid of 64 × 64, as well as models with time skips of 5 and 30 seconds. Finally, we fix the dataset such that we predict minutes 3 to 11, with the reasoning that the very early game is not as interesting since enemies have not been found, and any prediction might just be guessing, and the most interesting parts of the openings happen during this time frame. We choose not to use later than 11 minutes to avoid the long tail, since variability increases with game duration. We tried several different specifications of the model described in FIG2. We tried vanilla ConvNets, as well as one with enough striding such that the map is eventually downsampled into a 1 × 1 feature vector before R, thereby avoiding a pooling layer at the end of the encoder. We also try a conv-lstm encoder, where there are recurrent connections in the encoder. This is done by tiling the encoder recurrent layer one after the other E 1 → R 1 → E 2 → R 2 →...., where each encoder does some downsampling and each recurrent layer is replicated across the spatial grid. The recurrent layer (R) was always an LSTM, except for a conv-only model in which we replaced it with a temporal convolution. The decoder (D) was always another ConvNet, whose properties where other hyperparameters, with the same activation function (or residuals) as the encoder, without LSTMs ever. Its padding was set at all layers to be half the kernel size of the convolution rounded down, so that it produces exaclty one output tile per input tile. We did random search over the space of hyperparameters. We searched over two optimizers (SGD, BID6), the regression loss (MSE, Huber), λ values, and predicting the target value directly or predicting the difference between input and target, number of dimensions for hidden units, number of layers. We also searched over using residual blocks, gated convolutions, different nonlinearities. In our final models, we noticed several trends. Generally, gated convolutions BID2 did the best, while residual blocks with an information bottleneck did not improve over ReLU (probably due to the limited depth of our models). Adam was the more robust optimizer and converges faster than SGD, though SGD might converge to a better minimum given that it was trained for long enough. We noticed that the most robust (across tasks) model was the conv-lstm. The simple model performed the worst, as too much information is lost in the pooling layer after E (before R), and this layer is hard to avoid due to the difference in sizes of StarCraft maps. The striding model could learn, but is much more sensitive to hyperparameters, as was the conv-only model. Additionally, the lack of a memory in conv only make it much more difficult to reason about long term dependencies, as each frame of the model must reason about details such as the location of the enemy base, and therefore is less computationally efficient. In reference to section 4.1 and FIG2, we report with: L r Huber, L c binary cross entropy, R LSTM, D vanilla ConvNet, and two different flavors of E. For conv-lstm, we show for a model with depth d, meaning that there are d instances of a d-convolution layers followed by downsampling and a recurrent layer (so a d = 2 corresponds to 4 convolutions, d = 3 to 9). For striding, we show for a model with depth 4, entailing that the input is downsampled by a factor of 16, so in most cases there is no global pooling at the end. We define four metrics as proxies for measuring the impact of defogging in RTS games, we present some baselines and the of our models on these metrics, on a human dataset of StarCraft: Brood War games. We use the train, valid, and test set given by BID8 to do forward modeling and defogging. The dataset includes 65k human games of StarCraft: Brood War at generally high skill level. We implemented the models in PyTorch using TorchCraft BID16. We considered a variety of different metrics to measure the performance of our model on. Two downstream StarCraft tasks obvious for a forward model is strategy prediction and tactics prediction. In strategy prediction, presence or absence of certain buildings is central to determining what units the opponent will produce. Thus, we can measure the prediction of all opponent buildings in a future frame. In tactics prediction, the key is to determine the approximate locations of enemy units in the future. Furthermore, we care the most about enemy units that we cannot see. However, it's unclear how to define the fog of war when doing this prediction, since the areas we have vision might change drastically between two skip frames, so we measure the error on both visible and hidden enemy units. Thus, we measure two metrics, one where we define the fog of war to be just where the input frame does not see units, and one where we simply observe all enemy units. Finally, to use this in control, we wish to minimize the number of miscounts of enemy hidden units. This in 4 metrics. All metrics but the first (g_op_b) are measured per type per tile. When we measure existence/absence from a regression head, we take a threshold on the regression outputs so that numbers above this threshold counts as existence. These are cross validated per model. The metrics are: g_op_b (global opponent buildings) Existence of each opponent building type on any tile, from the classification head.hid_u (hidden units) Existence of hidden units, necessarily belonging to your opponent, from the regression head.op_u (opponent units) Existence of all opponent units output from the regression head, note that we see some of the opponent units (but they can move or be destroyed).abs_diff (absolute difference) An L1 loss on the counts of enemy units, from the regression head, over true positives Table 1: Score of the models, and of the best baseline for each task (in F1). See Table 2 in Appendix for full details. The absolute difference metric is measured only on the true positives of the model, so we subtract the best baseline to each model score, because a high precision low recall score will have a weaker cost and baseline cost than a high recall model. Thus, as the baseline is different for each model, to give an order of magnitude, we display the lowest (i.e. best of) the best baselines on the eponym line. These abs_diff numbers can only compare the models (who beat all the baselines as they are all negatives). More negative is better than baseline. * We could not train a single model to do well on both the regression and classification heads, so we display for op_b from a striding model with slightly different weights on each head. To validate the strength of our models, we validate their performance against some hard-coded baselines, similar to what rule-based bots use traditionally in StarCraft: Brood War competitions. These baselines rely exclusively on what was previously seen, as well as some rules of the games (to infer hidden requirements of units that we saw).We rely on four different baselines to measure success:1. Input -The baseline predicts by copying the input frame.2. Perfect memory (PM) -The baseline remembers everything in the past, and units are never removed, maximizing recall.3. Perfect memory + rules (PM+R) -This baseline is designed to maximize g_op_b, by using perfect memory and game rules to infer the existence of unit types that are prequisites for unit types that are seen.4. Previous Seen (PS) -This baseline takes the position of the last seen unit in the map, which is what most rule based bots do in real games. When a location is revealed and no units are at the spot, the count is again reset to 0.In order to beat these baselines, our models have to learn a good correlation model on the units and buildings type, remember what it has seen before and understand the dynamics of the movement of the units. We report baselines and models scores according to the metrics described above, on 64 and 32 walktiles effective grids (striding), with time steps of 5, 15, and 30 seconds, in Table 1.Figure 2: Plots show unit counts of the specified type per map cell (darker dots correspond to higher counts). Top row of each plot shows model inputs (i.e. observed units), middle row shows model predicted unit distributions and bottom row shows real unit distributions. In the left two plots, we observe that our model sees only a few opponent units, and tries to predict the location of their army under the for of war, in places it had not seen units before. In the right two plots, we observe our model remembering the locations of the opponent base and army. To obtain the existence thresholds from a regression output, we run a sweep on the validation set on 0.1 increments up to 1.5, where the model predicts existence if it predicts more than this value in the output. This value is usually very low, indicating that our model is sure of grid locations with zero units. We also finetune the probability of predicting global opponent building existence the same way, in that we predict existence if the probability output by the model is greater than p. Generally, p tends to be slightly above 0.5 to most maximize the F1 score. We report the on the test set with the best thresholds on the validation set. We note that for g_op_b prediction, the baselines already do very well. It is hard to beat the best baseline, PM+R. However, most of our models have higher recall than the baseline, indicating that they predict many more unexpected buildings, at the expense of mispredicting existing buildings. We do best above baseline on predicting the global existence of opponent buildings 30 seconds in the future, on whatever grid size. Our models make the most gains above baseline on unit prediction. Since units often move very erratically, this is difficult for a baseline that only predicts the previous frame. In order to predict units well, the model must have a good understanding of the dynamics of the game as well as the possible strategies taken by players in the game. For our baselines, the more coarse the grid size the easier it is to predict unit movement, since small jitters won't change the featurization. Additionally, predicting closer in time also improves the , since units move less often. We do 24% better in F1 in predicting the baseline for hidden opponent units 15 seconds in the future with a grid size of 32. In predicting all opponent units, we do 19% better, since baselines are able to more easily predict the existence of units that are already seen. These are the most useful cases for a defogger, since predict tactical movements 15 seconds in the future can aid a bot controller in predicting what opponents will do next. Predicting 5 seconds is not as useful, as enemies will not move much, and predicting at a grid size of 64 is harder for our model to use, as units on the diagonals of a grid box might not even see each other. We notice that our models have much higher precision than the baselines, with some over twice the precision. This supports the observation that our models have a better usage of their memory than the baselines, and are able to remember objects, but also "forget" them based on visibility of their previous positions and other correlates. On all unit prediction tasks, our models beat the baseline by a significant amount. Finally, we try to give a metric to approximate an estimate of how well the model can do in control. During control, we wish to minimize the number of mispredictions of opponent units. However, we noticed that the absolute number of mispredicted units would average to 10s of thousands per game for the baselines, and only several thousand per game for our model. This is because if we continually mispredict, for example, using the perfect memory baseline, then the mispredictions would add up over the length of the game. This shows how bad the baselines are at the regression task compared to classification, often 10x or more off from our models. To create more sane outputs comparable to the outputs of our models, we only display the L1 score over the true positives of the model. Table 2: Score of the models, and of the best baseline for each task (in F1). The absolute difference metric is measured only on the true positives of the model, so we subtract the best baseline. Thus, the baseline is different for each model, so we do not display it. More negative is better than baseline.

This paper presents a defogger, a model that learns to predict future hidden information from partial observations, applied to a StarCraft dataset.

represent the tldr for finding the corresponding scientific article.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this work we study generalization of neural networks in gradient-based meta-learning by analyzing various properties of the objective landscapes. We experimentally demonstrate that as meta-training progresses, the meta-test solutions obtained by adapting the meta-train solution of the model to new tasks via few steps of gradient-based fine-tuning, become flatter, lower in loss, and further away from the meta-train solution. We also show that those meta-test solutions become flatter even as generalization starts to degrade, thus providing an experimental evidence against the correlation between generalization and flat minima in the paradigm of gradient-based meta-leaning. Furthermore, we provide empirical evidence that generalization to new tasks is correlated with the coherence between their adaptation trajectories in parameter space, measured by the average cosine similarity between task-specific trajectory directions, starting from a same meta-train solution. We also show that coherence of meta-test gradients, measured by the average inner product between the task-specific gradient vectors evaluated at meta-train solution, is also correlated with generalization. To address the problem of the few-shot learning, many meta-learning approaches have been proposed recently , , , and among others. In this work, we take steps towards understanding the characteristics of the landscapes of the loss functions, and their relation to generalization, in the context of gradient-based few-shot meta-learning. While we are interested in understanding the properties of optimization landscapes that are linked to generalization in gradient-based meta-learning in general, we focus our experimental work here within a setup that follows the recently proposed Model Agnostic Meta-Learning (MAML) algorithm . The MAML algorithm is a good candidate for studying gradient-based meta-learning because of its independence from the underlying network architecture. Our main insights and contributions can be summarized as follows: 1. As gradient-based meta-training progresses: • the adapted meta-test solutions become flatter on average, while the opposite occurs when using a finetuning baseline. • the adapted final solutions reach lower average support loss values, which never increases, while the opposite occurs when using a finetuning baseline. 2. When generalization starts to degrade due to overtraining, meta-test solutions keep getting flatter, implying that, in the context of gradient-based meta-learning, flatness of minima is not correlated with generalization to new tasks. 3. We empirically show that generalization to new tasks is correlated with the coherence between their adaptation trajectories, measured by the average cosine similarity between trajectory directions. Also correlated with generalization is the coherence between metatest gradients, measured by the average inner product between meta-test gradient vectors evaluated at meta-train solution. We also show that this metric is correlated to generalization for few-shot regression tasks where the model must learn to fit sine function curves. Furthermore, based on these observations, we take initial steps to propose a regularizer for MAML based training and provide experimental evidence for its effectiveness. There has been extensive research efforts on studying the optimization landscapes of neural networks in the standard supervised learning setup. Such work has focused on the presence of saddle points versus local minima in high dimensional landscapes,, the role of overparametrization in generalization , loss barriers between minima and their connectivity along low loss paths, ; , to name a few examples. One hypothesis that has gained popularity is that the flatness of minima of the loss function found by stochastic gradient-based methods in good generalization, ; . and measure the flatness by the spectral norm of the hessian of the loss, with respect to the parameters, at a given point in the parameter space. Both and consider the determinant of the hessian of the loss, with respect to the parameters, for the measure of flatness. For all of the work on flatness of minima cited above, authors have found that flatter minima correlate with better generalization. In contrast to previous work on understanding the objective landscapes of neural networks in the classical supervised learning paradigm, in our work, we explore the properties of objective landscapes in the setting of gradient-based meta-learning. We consider the meta-learning scenario where we have a distribution over tasks p(T), and a model f parametrized by θ, that must learn to adapt to tasks T i sampled from p(T). The model is trained on a set of training tasks {T i} train and evaluated on a set of testing tasks {T i} test, all drawn from p(T). In this work we only consider classification tasks, with {T i} train and {T i} test using disjoint sets of classes to constitute their tasks. Here we consider the setting of k-shot learning, that is, when f adapts to a task T test i, it only has access to a set of few support samples. We then evaluate the model's performance on T test i using a new set of target samples D i. By gradient-based meta-learning, we imply that f is trained using information about the gradient of a certain loss function L(f (D i ; θ)) on the tasks. Throughout this work the loss function is the cross-entropy between the predicted and true class. MAML learns an initial set of parameters θ such that on average, given a new task T order approximation of MAML, where these second-order derivatives are ommited, and we refer to that other algorithm as First-Order MAML. For the finetuning baseline, the model is trained in a standard supervised learning setup: the model is trained to classify all the classes from the training split using a stochastic gradient-based optimization algorithm, its output layer size being equal to the number of meta-train classes. During evaluation on meta-test tasks, the model's final layer (fully-connected) is replaced by a layer with the appropriate size for the given meta-test task (e.g. if 5-way classification, the output layer has five logits), with its parameter values initialized to random values or with another initialization algorithm, then all the model parameters are optimized to the meta-test task, just like for the other meta-learning algorithms. Figure 1: Visualizations of metrics measuring properties of objective loss landscapes. The black arrows represent the descent on the support loss and the dotted lines represent the corresponding displacement in the parameter space.: Curvature of the loss for an adapted meta-test solutionθi (for a task Ti), is measured as the spectral norm of the hessian matrix of the loss.: Coherence of adaptation trajectories to different meta-test tasks is measured as the average cosine similarity for pairs of trajectory directions. A direction vector is obtained by dividing a trajectory displacement vector (from meta-train solution θ s to meta-test solutioñ Characterizing a meta-train solution by the coherence of the meta-test gradients, measured by the average inner product for pairs of meta-test gradient In the context of gradient-based meta-learning, we define generalization as the model's ability to reach a high accuracy on a testing task T test i, evaluated with a set of target samples D i, for several testing tasks. This accuracy is computed after f, starting from a given meta-training parametrization θ s, has optimized its parameters to the task T, we consider the optimization landscapes L(f (D i ; θ)), and 1) the properties of these loss landscapes evaluated at the solutionsθ test i; 2) the adaptation trajectories when f, starting from θ s, adapts to those solutions; as well as 3) the properties of those landscapes evaluated at the meta-train solutions θ s. See Figure 1 for a visualization of our different metrics. We follow the evolution of the metrics as meta-training progresses: after each epoch, which in a different parametrization θ s, we adapt f to several meta-test tasks, compute the metrics averaged over those tasks, and compare with We do not deal with the objective landscapes involved during meta-training, as this is beyond the scope of this work. From here on, we drop the superscript test from our notation, as we exclusively deal with objective landscapes involving meta-test tasks T i, unless specified otherwise. We start our analysis of the objective loss landscapes by measuring properties of the landscapes at the adapted meta-test solutionsθ i. More concretely, we measure the curvature of the loss at those minima, and whether flatter minima are indicative of better generalization for the meta-test tasks. After s meta-training iterations, we have a model f parametrized by θ s. During the meta-test, f must adapt to several meta-test tasks T i independently. For a given T i, f adapts by performing a few steps of full-batch gradient descent on the objective landscape L(f (D i ; θ) ), using the set of support samples D i, and reaches an adapted solutionθ i. Here we are interested in the curvature of L(f (D i ;θ i)), that is, the objective landscape when evaluated at such solution, and whether on average, flatter solutions favour better generalization. Considering the hessian matrix of this loss w.r.t the model parameters, defined as, we measure the curvature of the loss surface aroundθ i using the spectral norm · σ of this hessian matrix: as illustrated in Figure 1 (We define the average loss curvature for meta-test solutionsθ i, obtained from a meta-train solution θ s, as: Note that we do not measure curvature of the loss at θ s, since θ s is not a point of convergence of f for the meta-test tasks. In fact, at θ s, since the model has not been adapted to the unseen meta-test classes, the target accuracy for the meta-test tasks is random chance on average. Thus, measuring the curvature of the meta-test support loss at θ s does not relate to the notion of flatness of minima. Instead, in this work we characterize the meta-train solution θ s by measuring the average inner product between the meta-test gradients, as explained later in Section 4.3. Other than analyzing the objective landscapes at the different minima reached when f adapts to new tasks, we also analyze the adaptation trajectories to those new tasks, and whether some similarity between them can be indicative of good generalization. Let's consider a model f adapting to a task T i by starting from θ s, moving in parameter space by performing T steps of full-batch gradient descent with ∇ θ L(f (D i ; θ)) until reachingθ i. We define the adaptation trajectory to a task T i starting from θ s as the sequence of iterates (θ s, θ i, ...,θ i). To simplify the analyses and alleviate some of the challenges in dealing with trajectories of multiple steps in a parameter space of very high dimension, we define the trajectory displacement vector (θ i − θ s). We define a trajectory direction vector θ i as the unit vector: We define a metric for the coherence of adaptation trajectories to meta-test tasks T i, starting from a meta-train solution θ s, as the average inner product between their direction vectors: The inner product between two meta-test trajectory direction vectors is illustrated in Figure 1. In addition to characterizing the adaptation trajectories at meta-test time, we characterize the objective landscapes at the meta-train solutions θ s. More concretely, we measure the coherence of the meta-test The coherence between the meta-test gradients can be viewed in relation to the metric for coherence of adaptation trajectories of Eq. 5 from Section 4.2. Even after simplifying an adaptation trajectory by its displacement vector, measuring distances between trajectories of multiple steps in the parameter space can be problematic: because of the symmetries within the architectures of neural networks, where neurons can be permuted, different parameterizations θ can represent identically the same function f that maps inputs to outputs. This problem is even more prevalent for networks with higher number of parameters. Since here we ultimately care about the functional differences that f undergoes in the adaptation trajectories, measuring distances between functions in the parameter space, either using Euclidean norm or cosine similarity between direction vectors, can be problematic . Thus to further simplify the analyses on adaptation trajectories, we can measure coherence between trajectories of only one step (T = 1). Since we are interested in the relation between such trajectories and the generalization performance of the models, we measure the target accuracy at those meta-test solutions obtained after only one step of gradient descent. We define those solutions as: To make meta-training consistent with meta-testing, for the meta-learning algorithms we also use T = 1 for the inner loop updates of Eq. 1. We thus measure coherence between the meta-test gradient vectors g i that lead to those solutions. Note that the learning rate α is constant and is the same for all experiments on a same dataset. In contrast to Section 4.2, here we observed in practice that the average inner product between meta-test gradient vectors, and not just their direction vectors, is more correlated to the average target accuracy. The ing metric is thus the average inner product between meta-test gradients evaluated at θ s. We define the average inner product between meta-test gradient vectors g i, evaluated at a meta-train solution θ s, as: The inner product between two meta-test gradients, evaluated at θ s, is illustrated in Figure 1. We show in the experimental in Section 5.2 and 5.3 that the coherence of the adaptation trajectories, as well as of the meta-test gradients, correlate with generalization on the meta-test tasks. We apply our analyses to the two most widely used benchmark datasets for few-shot classification problems: Omniglot and MiniImagenet datasets. We use the standardized CNN architecture used by and . We perform our experiments using three different gradient-based meta-learning algorithms: MAML, First-Order MAML and a Finetuning baseline. For more details on the meta-learning datasets, architecture and meta-learning hyperparameters, see Appendix A We closely follow the experimental setup of . Except for the Finetune baseline, the meta-learning algorithms use during meta-training the same number of ways and shots as during metatesting. For our experiments, we follow the setting of : for MiniImagenet, training and testing our models on 5-way classification 1-shot learning, as well as 5-way 5-shot, and for Omniglot, 5-way 1-shot; 5-way 5-shot; 20-way 1-shot; 20-way 5-shot. Each experiment was repeated for five independent runs. For the meta-learning algorithms, the choice of hyperparameters closely follows . For our finetuning baseline, most of the original MAML hyperparameters were left unchanged, as we want to compare the effect of the pre-training procedure, thus are kept fixed the architecture and meta-test procedures. We kept the same optimizer as for the meta-update of MAML (ADAM), and performed hyperparameter search on the mini-batch size to use, for each setting that we present. (For our reproduction on the meta-train and meta-test accuracy, see Figure 10a and 10b in B.1.) After each training epoch, we compute E[H θ (D i ;θ i) σ ] using a fixed set of 60 randomly sampled meta-test tasks T i. Across all settings, we observe that MAML first finds sharper solutionsθ i until reaching a peak, then as the number of epoch grows, those solutions become flatter, as seen in Figure 2. To verify the correlation between On the contrary, and remarkably, even as f starts to show poorer generalization (see Figure 3a), the solutions keep getting flatter, as shown in Figure 3c. Thus for the case of gradient-based meta-learning, flatter minima don't appear to favour better generalization. We perform the same analysis for our finetuning baseline (Figures 4a, 4c), with suggesting that flatness of solutions might be more linked with E[L(f (D i ;θ i))], the average level of support loss attained by the solutionsθ i (see Figures 4b and 3b), which is not an indicator for generalization. We also noted that across all settings involving MAML and First-Order MAML, this average meta-test support loss E[L(f (D i ;θ i))] decreases monotonically as meta-training progresses. In this section, we use the same experimental setup as in Section 5.1, except here we measure To reduce the variance on our , we sample 500 tasks after each meta-training epoch. Also for experiments on Omniglot, we drop the analyses with First-Order MAML, since it yields performance very similar to that of the Second-Order MAML. We start our analyses with the setting of "MiniImagenet, First-Order MAML, 5-way 1-shot", as it allowed us to test and invalidate the correlation between flatness of solutions and generalization, earlier in Section 5.1. We clearly observe a correlation between the coherence of adaptation trajectories and generalization to new tasks, with higher average inner product between trajectory directions, thus smaller angles, being linked to higher average target accuracy on those new tasks, as shown in Figure 5a. We then performed the analysis on the other settings, with the same observations (see Figure 5b and Figure 11 in Appendix B.2 for full set of experiments). We also perform the analysis on the Finetuning baselines, which reach much lower target accuracies, and where we see that E[θ T i θ j] remains much closer to zero, meaning that trajectory directions are roughly orthogonal to each other, akin to random vectors in high dimension (see Figure 6a). As an added observation, here we include our experimental on the average meta-test trajectory norm E[θ i − θ Despite the clear correlation between E[ θ T i θ j] and generalization for the settings that we show in Figure 5 and 11, we observed that for some other settings, this relationship appears less linear. We conjecture that such behavior might arise from the difficulties of measuring distances between networks in the parameter space, as explained in Section 4.3. Here we present our on the characterization of the objective landscapes at the meta-train solutions θ s, by measuring the average inner product between meta-test gradient vectors g i. We observe that coherence between meta-test gradients is correlated to generalization, which is consistent with the observations on the coherence of adaptation trajectories from Section 5.2. In Figure 7, we compare E[g experiments. This metric consistently correlates with generalization across the different settings. Similarly as in Section 5.2, for our finetuning baselines we observe very low coherence between meta-test gradients (see Figure 6b). Based on the observations we make in Section 5.2 and 5.3, we propose to regularize gradient-based meta-learning as described in Section 6. Here we extend our analysis by presenting experimental on E[g T i g j] for few-shot regression. Specifically we use a leaning problem which is composed of training task and test tasks, where each of these tasks are sine functions parameterized as y = a sin(bx + c). We train a two-layer MLP which learns to fit meta-training sine functions using only few support samples, and generalization implies reaching a low Mean Squared Error (MSE) averaged over the target set of many meta-test sine functions. Results are presented in Figure 8. Similar to our analysis of Few-shot classification setting, we observe in the case of Few-shot regression, generalization (negative average target MSE on Meta-test Task) strongly correlates with E[g Although, MAML has become a popular method for meta-training, there exist a significant generalization gap between its performance on target set of the meta-train tasks and the target set of the meta-test task, and regularizing MAML has not received much research attention yet. Based on our observations on the coherence of adaptation trajectories, we take first steps in this direction by adding a regularization term based on E[ θ T i θ j]. Within a meta-training iteration, we first let f adapt to the n training tasks T i following Eq 1. We then compute the average direction vector θ µ = 1 n n i=1 θ i. For each task, we want to reduce the angle defined by θ T i θ µ, and thus introduce the penalty on T i θ µ, obtaining the regularized solutionsθ i. The outer loop gradients are then computed, just like in MAML following Eq 2, but using these regularized solutionsθ i instead ofθ i. We obtain the variant of MAML with regularized inner loop updates, as detailed in Algorithm 1. We used this regularizer with MAML (Second-Order), for "Omniglot 20-way 1-shot", thereby tackling the most challenging few-shot classification setting for Omniglot. As shown in Figure 9, we observed an increase in meta-test target accuracy: the performance increases from 94.05% to 95.38% (average over five trials, 600 test tasks each), providing ∼ 23% relative reduction in meta-test target error. Algorithm 1 Regularized MAML: Added penalty on angles between inner loop updates 1: Sample a batch of n tasks Perform inner loop adaptation as in Eq. 1: i )) 4: end for 5: Compute the average direction vector: Compute the corrected inner loop updates: 7: for all T i do 8:θ i =θ i −γ∇ θ Ω(θ) where Ω(θ) = − θ T i θ µ 9: end for 10: Perform the meta-update as in Eq. 2, but using the corrected solutions: We experimentally demonstrate that when using gradient-based meta-learning algorithms such as MAML, meta-test solutions, obtained after adapting neural networks to new tasks via few-shot learning, become flatter, lower in loss, and further away from the meta-train solution, as metatraining progresses. We also show that those meta-test solutions keep getting flatter even when generalization starts to degrade, thus providing an experimental argument against the correlation between generalization and flat minima. More importantly, we empirically show that generalization to new tasks is correlated with the coherence between their adaptation trajectories, measured by the average cosine similarity between the adaptation trajectory directions, but also correlated with the coherence between the meta-test gradients, measured by the average inner product between meta-test gradient vectors evaluated at meta-train solution. We also show this correlation for few-shot regression tasks. Based on these observations, we take first steps towards regularizing MAML based meta-training. As a future work, we plan to test the effectiveness of this regularizer on various datasets and meta-learning problem settings, architectures and gradient-based meta-learning algorithms. A ADDITIONAL EXPERIMENTAL DETAILS We use the architecture proposed by which is used by , consisting of 4 modules stacked on each other, each being composed of 64 filters of of 3 × 3 convolution, followed by a batch normalization layer, a ReLU activation layer, and a 2 × 2 maxpooling layer. With Omniglot, strided convolution is used instead of max-pooling, and images are downsampled to 28 × 28. With MiniImagenet, we used fewer filters to reduce overfitting, but used 48 while MAML used 32. As a loss function to minimize, we use cross-entropy between the predicted classes and the target classes. The Omniglot dataset consists of a total of 1623 classes, each comprising 20 instances. The classes correspond to distinct characters, taken from 50 different datasets, but the taxonomy among characters isn't used. The MiniImagenet dataset comprises 64 training classes, 12 validation classes and 24 test classes. Each of those classes was randomly sampled from the original Imagenet dataset, and each contains 600 instances with a reduced size of 84 × 84. We follow the same experimental setup as for training and testing the models using MAML and First-Order MAML. During meta-training, the inner loop updates are performed via five steps of full batch gradient descent (except for Section 5.3 where T = 1), with a fixed learning rate α of 0.1 for Omniglot and 0.01 for MiniImagenet, while ADAM is used as the optimizer for the meta-update, without any learning rate scheduling, using a meta-learning rate β of 0.001. At meta-test time, adaptation to meta-test task is always performed by performing the same number of steps as for the meta-training inner loop updates. We use a mini-batch of 16 and 8 tasks for the 1-shot and 5-shot settings respectively, while for the MiniImagenet experiments, we use batches of 4 and 2 tasks for the 1-shot and 5-shots settings respectively. Let's also precise that, in k-shot learning for an m-way classification task T i, the set of support samples D i comprises k × m samples. Each meta-training epoch comprises 500 meta-training iterations. For the finetuning baseline, we kept the same hyperparameters for the ADAM optimizer during meta-training, and for the adaptation during meta-test. We searched the training hyperparameter values for the mini-batch size and the number of iterations per epoch. Experiments are run for a 100 epochs each. In order to limit meta-overfitting and maximize the highest average meta-test target accuracy, the finetuning models see roughly 100 times less training data per epoch compared to a MAML training epoch. In order to evaluate the baseline on the 1-shot and 5-shot meta-test tasks, during training we used mini-batches of 64 images with 25 iterations per epoch for 1-shot learning, and mini-batches of 128 images with 12 iterations per epoch, for 5-shot learning. At meta-test time, we use Xavier initialization to initialize the weights of the final layer. For the few-shot regression problems (which is also present in the work of ( The performance of the models trained with MAML and First-Order MAML, for the few-shot learning settings of Omniglot and MiniImagenet, are presented in Figure 10 . They include the target accuracies on meta-train tasks and on meta-test tasks (generalization), as meta-training progresses. The relation between target accuracy on meta-test tasks, and angles between trajectory directions is presented in Figure 11. The relation between target accuracy on meta-test tasks, and average inner product between meta-test gradients evaluated at meta-train solution, is presented in Figure 12.

Represent the short summary for finding the corresponding scientific article (long).

We study generalization of neural networks in gradient-based meta- learning by analyzing various properties of the objective landscape.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: There have been multiple attempts with variational auto-encoders (VAE) to learn powerful global representations of complex data using a combination of latent stochastic variables and an autoregressive model over the dimensions of the data. However, for the most challenging natural image tasks the purely autoregressive model with stochastic variables still outperform the combined stochastic autoregressive models. In this paper, we present simple additions to the VAE framework that generalize to natural images by embedding spatial information in the stochastic layers. We significantly improve the state-of-the-art on MNIST, OMNIGLOT, CIFAR10 and ImageNet when the feature map parameterization of the stochastic variables are combined with the autoregressive PixelCNN approach. Interestingly, we also observe close to state-of-the-art without the autoregressive part. This opens the possibility for high quality image generation with only one forward-pass. In representation learning the goal is to learn a posterior latent distribution that explains the observed data well BID0. Learning good representations from data can be used for various tasks such as generative modelling and semi-supervised learning (; BID14 BID14 BID23 . The decomposition of variational auto-encoders (VAE) (; BID14 provides the potential to disentangle the internal representation of the input data from local to global features through a hierarchy of stochastic latent variables. This makes the VAE an obvious candidate for learning good representations. However, in order to make inference tractable VAEs contain simplifying assumptions. This limits their ability to learn a good posterior latent representation. In complex data distributions with temporal dependencies (e.g. text, images and audio), the VAE assumption on conditional independence in the input distribution limits the ability to learn local structures. This has a significant impact on its generative performance, and thereby also the learned representations. Additionally, the one-layered VAE model with a N (0, I) latent prior poses serious constraints on the posterior complexity that the model is able to learn. A deep hierarchy of stochastic latent variables should endow the model with more expressiveness, but the VAE has a tendency to skip the learning of the higher representations since they pose a direct cost in its optimization term. There have been several attempts to eliminate the limitations of the VAE. Some concern formulating a more expressive variational distribution BID3 BID25 BID30 where other concerns learning a deeper hierarchy of latent variables. These contributions have ed in better performance, but are still limited when modelling complex data distributions where a conditional independence does not apply. When parameterizing the VAE decoder with recurrent neural networks BID17 BID1 BID7, the decoding architecture gets too powerful which in unused latent stochastic variables.The limitations of the VAE have spawned interest towards other generative models such as Generative Adversarial Networks (GAN) BID8 and the autoregressive Pixel-CNN/PixelRNN models BID33. These methods have proven powerful in learning good generative models, but the lack of stochastic latent variables makes them less suitable for representation learning purposes. Lately, we have seen several successful attempts to combine VAEs with PixelCNNs BID11. This Figure 1: A visualization of FAME where the solid lines denote the variational approximation (inference/encoder/recognition) network and dashed lines denote the generative model (decoder) network for training. When performing reconstructions during training, the input image is concatenated with the output of the generative model (blue) and when generating the model follows a normal autoregressive sampling flow (red) while also using the stochastic latent variables z = z 1,..., z L. Both the variational approximation and the generative model follow a top-down hierarchical structure which enables precision weighted stochastic variables in the variational approximation.in a model where the global structure of the data is learned in the stochastic latent variables of the VAE and the local structure is learned in the PixelCNN. However, despite the additional complexity and potential extra expressiveness, these models do not outperform a simple autoregressive model BID32.In this paper we present the Feature Map Variational Auto-Encoder (FAME) that combines the top-down variational approximation presented in the Ladder Variational Auto-Encoder (LVAE) ) with a spatial (feature map) representation of the stochastic latent variables and an autoregressive decoder. We show that (i) FAME outperforms previously state-of-the-art loglikelihood on MNIST, OMNIGLOT, CIFAR10 and ImageNet, (ii) FAME learns a deep hierarchy of stochastic latent variables without inactivated latent units, (iii) by removing the autoregressive decoder FAME performs close to previous state-of-the-art log-likelihood suggesting that it is possible to get good quality generation with just one forward pass. The VAE BID14 ) is a generative model with a hierarchy of stochastic latent variables: DISPLAYFORM0 where z = z 1,..., z L, θ denotes the parameters, and L denotes the number of stochastic latent variable layers. The stochastic latent variables are usually modelled as conditionally independent Gaussian distributions with a diagonal covariance: DISPLAYFORM1 Since the posterior p(z|x) often is intractable we introduce a variational approximation q φ (z|x) with parameters φ. In the original VAE formulation q φ (z|x) is decomposed as a bottom-up inference path through the hierarchy of the stochastic layers: DISPLAYFORM2 DISPLAYFORM3 DISPLAYFORM4 We optimize an evidence lower-bound (ELBO) to the log-likelihood log p θ (x) = log z p θ (x, z)dz. BID2 introduced the importance weighted bound: DISPLAYFORM5 and proved that DISPLAYFORM6 For K = 1 the bound co-incides with the standard ELBO: L(θ, φ; x) = L 1 (θ, φ; x). The hierarchical structure of both the variational approximation and generative model give the VAE the expressiveness to learn different representations of the data throughout its stochastic variables, going from local (e.g. edges in images) to global features (e.g. class specific information). However, we can apply as recursive argument BID22 to show that when optimizing with respect to the parameters θ and φ the VAE is regularized DISPLAYFORM7. This is evident if we rewrite Equation 6 for K = 1: DISPLAYFORM8 is a local maxima and learning a useful representation in z L can therefore be disregarded throughout the remainder of the training. The same argumentation can be used for all subsequent layers z 2:L, hence the VAE has a tendency to collapse towards not using the full hierarchy of latent variables. There are different ways to get around this tendency, where the simplest is to down-weight the KL-divergence with a temperature term BID1. This term is applied during the initial phase of optimization and thereby downscales the regularizing effect. However, this only works for a limited number of hierarchically stacked latent variables.Formulating a deep hierarchical VAE is not the only cause of inactive latent variables, it also occurs when the parameterization of the decoder gets too powerful BID17 BID7. This can be caused by using autoregressive models such as p(circumvent this by introducing the Variational Lossy AutoEncoder (VLAE) where they define the architecture for the VAE and autoregressive model such that they capture global and local structures. They also utilize the power of more expressive posterior approximations using inverse autoregressive flows BID25 BID15. In the PixelVAE, BID11 takes a similar approach to defining the generative model but makes a simpler factorizing decomposition in the variational approximation q φ (z|x) = L i q φ (z i |x), where the terms have some degree of parameter sharing. This formulation in a less flexible model. DISPLAYFORM9 In BID15; BID11 we have seen that VAEs with simple decompositions of the stochastic latent variables and a powerful autoregressive decoder can in good generative performance and representation learning. However, despite the additional cost of learning a VAE we only see improvement in the log-likelihood over the PixelCNN for small gray-scale image datasets. We propose FAME that extends the VAE with a top-down variational approximation similar to the LVAE (Sønderby et al., 2016) combined with spatial stochastic latent layers and an autoregressive decoder, so that we ensure expressive latent stochastic variables learned in a deep hierarchy (cf. Figure 1). The LVAE does not change the generative model but changes the variational distribution to be top-down like the generative model. Furthermore the variational distribution shares parameters with the generative model which can be viewed as a precision-weighted (inverse variance) combination of information from the prior and data distribution. The variational approximation is defined as: DISPLAYFORM0 The stochastic latent variables are all fully factorized Gaussian distributions and are therefore modelled by q φ (z i |z i+1, x) = N (z i |µ i, diag(σ separate parameters (as in the VAE), the LVAE let the mean and variance be defined in terms of a function of x (the bottom-up data part) and the generative model (the top-down prior): DISPLAYFORM1 DISPLAYFORM2 where µ φ,i = µ φ,i (x) and µ θ,i = µ θ,i (z i+1) and like-wise for the variance functions. This precision weighted parameterization has previously yielded excellent for densely connected networks. We have seen multiple contributions (e.g. BID11) where VAEs (and similar models) have been parameterized with convolutions in the deterministic layers h i j, for j = 1,..., M, and M is the number of layers connecting the stochastic latent variables z i. The size of the spatial feature maps decreases towards higher latent representations and transposed convolutions are used in the generative model. In FAME we propose to extend this notion, so that each of the stochastic latent layers z i,..., z L−1 are also convolutional. This gives the model more expressiveness in the latent layers, since it will keep track of the spatial composition of the data (and thereby learn better representations). The top stochastic layer z L in FAME is a fully-connected dense layer, which makes it simpler to condition on a non-informative N (0, I) prior and sample from a learned generative model p θ (x, z). For the i = 1,..., L − 1 stochastic latent variables, the architecture is as follows: DISPLAYFORM0 where CNN and CONV denote a convolutional neural network and convolutional layer respectively. The top-most latent stochastic layer z L is computed by: DISPLAYFORM1 This new feature map parameterization of the stochastic layers should be viewed as a step towards a better variational model where the test ELBO and the amount of activated stochastic units are direct meaures hereof. From van den Oord et al. FIG0; we have seen that the PixelCNN architecture is very powerful in modelling a conditional distribution between pixels. In FAME we introduce a PixelCNN in the input dimension of the generative model p θ (x|z) (cf. Figure 1). During training we concatenate the input with the reconstruction data in the channel dimension and propagate it through the PixelCNN, similarly to what is done in BID11. When generating samples we fix a sample from the stochastic latent variables and generate the image pixel by pixel autoregressively. We test FAME on images from which we can compare with a wide range of generative models. First we evaluate on gray-scaled image datasets: statically and dynamically binarized MNIST and OMNIGLOT . The OMNIGLOT dataset is of particular interest due to the large variance amongst samples. Secondly we evaluate our models on natural image datasets: CIFAR10 BID18 Table 2: Negative log-likelihood performance on dynamically (left) and statically (right) binarized MNIST in nats. For the dynamically binarized MNIST show the for the FAME No Concatenation that has no dependency on the input image. The evidence lower-bound is computed with 5000 importance weighted samples L 5000 (θ, φ; x).modelling the gray-scaled images we assume a Bernoulli B distribution using a Sigmoid activation function as the output and for the natural images we assume a Categorical distribution π by applying the 256-way Softmax approach introduced in van den BID33. We evaluate the grayscaled images with L 5000 (cf. Equation 6) and due to runtime and space complexity we evaluate the natural images with L 1000.We use a hierarchy of 5 stochastic latent variables. In case of gray-scaled images the stochastic latent layers are dense with sizes 64, 32, 16, 8, 4 (equivalent to Sønderby et al. FORMULA0) and for the natural images they are spatial (cf. Table 1). There was no significant difference when using feature maps (as compared to dense layers) for modelling gray-scaled images. We apply batchnormalization BID12 and ReLU activation functions as the non-linearity between all hidden layers h i,j and use a simple PixelCNN as in van den BID33 with 4 residual blocks. Because of the concatenation in the autoregressive decoder (cf. Figure 1), generation is a cumbersome process that scales linearly with the amount of pixels in the input image. Therefore we have defined a slightly changed parameterization denoted FAME No Concatenation, where the concatenation with the input is omitted. The generation has no dependency on the input data distribution and can therefore be performed in one forward-pass through the generative model. For optimization we apply the Adam optimizer with a constant learning rate of 0.0003. We use 1 importance weighted sample and temperature scaling from.3 to 1. during the initial 200 epochs for gray-scaled images and.01 to 1. during the first 400 epochs for natural images. All models are trained using the same optimization scheme. The MNIST dataset serves as a good sanity check and has a myriad of previously published generative modelling benchmarks. We experienced much faster convergence rate on FAME compared to training a regular LVAE. On the dynamically binarized MNIST dataset we see a sig- Table 1: The convolutional layer (Conv), filter size (F), depth (K), stride (S), dense layer (Dense) and dimensionality (D) used in defining FAME for gray-scaled and natural images. The architecture is defined such that we ensure dimensionality reduction throughout the hierarchical stochastic layers. The autoregressive decoder is a PixelCNN (van den b) with a mask A convolution F=7x7, K=64, S=1 followed by 4 residual blocks of convolutions with mask B, F=3x3, K=64, S=1.Finally there are three non-residual layers of convolutions with mask B where the last is the output layer with a Sigmoid activation for gray-scaled images and a 256-way Softmax for natural images.nificant improvement (cf. Table 2). However, on the statically binarized MNIST, the parameterization and current optimization strategy was unsuccessful in achieving state-of-the-art (cf. Table 1). In FIG1 we see random samples drawn from a N (0, I) distribution and propagated through the decoder parameters θ. We also trained the FAME No Concatenation which performs nearly on par with the previously state-of-the-art VLAE model ) that in comparison utilizes a skip-connection from the input distribution to the generative decoder: DISPLAYFORM0. This proves that a better parameterization of the VAE improves the performance without the need of tedious autoregressive generation. There was no significant difference in the KL q(z|x)||p(z) between FAME and FAME No Concatenation. FAME use 10.85 nats in average to encode images, whereas FAME No Concatenation use 12.29 nats. NLL IWAE (BURDA ET AL., 2015A) 103.38 LVAE (SØNDERBY 102.11 RBM (BURDA ET AL., 2015B) 100.46 DVAE BID26 97.43 DRAW (GREGOR 96.50 CONV DRAW (GREGOR ET AL., 2016) 91.00 VLAE CHEN 89.83 FAME 82.54Figure 3: Negative log-likelihood performance on OMNIGLOT in nats. The evidence lower-bound is computed with 5000 importance weighted samples L 5000 (θ, φ; x).OMNIGLOT consists of 50 alphabets of handwritten characters, where each character has a limited amount of samples. Each character has high variance which makes it harder to fit a good generative model compared to MNIST. TAB3 presents the negative log-likelihood of FAME for OMNIGLOT and demonstrates significant improvement over previously published state-of-the-art. FIG1 shows generated samples from the learned θ parameter space. From Sønderby et al. FORMULA0 we have seen that the LVAE is able to learn a much tighter L 1 ELBO compared to the VAE. For the MNIST experiments, the L 1 ELBO is at 80.11 nats compared to the L 5000 77.82 nats. Similarly the OMNIGLOT L 1 ELBO is 86.62 nats compared to 82.54 nats. This shows significant improvements when using importance weighted samples and indicates that the parameterization of the FAME can be done in a way so that the bound is even tighter. We also find that the top-most latent stochastic layer is not collapsing into its prior, since the KL q(z 5 |x)||p(z 5) is 5.04 nats for MNIST and 3.67 nats for OMNIGLOT. In order to analyze the contribution from the autoregressive decoder we experimented on masking the contribution from either the concatenated image or the output of the FAME decoder before feeding it into the PixelCNN layers (cf. Figure 1). In FIG2 we see the of reconstructing MNIST images when masking out the contribution from the stochastic variables and in FIG2 we mask out the contribution from the concatenated input image. We investigate the performance of FAME on two natural image datasets: CIFAR10 and ImageNet. Learning a generative model on natural images is more challenging, which is also why there are many tricks that can be done in regards to the autoregressive decoding BID32. However, since we are interested in the additional expressiveness of a LVAE parameterization with convolutional stochastic latent variables, we have chosen a suboptimal architecture for the autoregressive decoding (cf. Table 1) BID33 ). An obvious improvement to the decoder would be to incorporate the PixelCNN++, but by using the simpler architecture we ensure that the improvements in log-likelihood is not a of a strong autoregressive model. From TAB3 we see the performance from FAME and FAME No Concatenation on the CIFAR10 dataset. Similarly to the gray-scaled images, FAME outperforms current state-of-the-art sig- Table 4: Negative log-likelihood performance on ImageNet in bits/dim. The evidence lower-bound is computed with 1000 importance weighted samples L 1000 (θ, φ; x).nificantly. It is also interesting to see how FAME No Concatenation performs close to the previously published state-of-the-art . Especially in the image space, this could prove interesting, since the FAME No Concatenation has no additional autoregressive runtime complexity. We only investigated the 32x32 ImageNet dataset, since the training time is significant and it outperformed the 64x64 models (cf. Table 4), whereas the previously published 64x64 ImageNet models consistently outperform their 32x32 counterpart. In FIG0 we show samples from FAME on the CIFAR10 dataset. Similarly to previously published it is difficult to analyze the performance from the samples. However, we can conclude that FAME is able to capture spatial correlations in the images for generating sharp samples. It is also interesting to see how it captures the contours of objects in the images. We have presented FAME, an extension to the VAE that significantly improve state-of-the-art performance on standard benchmark datasets. By introducing feature map representations in the latent stochastic variables in addition to top-down inference we have shown that the model is able to capture representations of complex image distributions while utilizing a powerful autoregressive architecture as a decoder. In order to analyze the contribution from the VAE as opposed to the autoregressive model, we have presented without concatenating the input image when reconstructing and generating. This parameterization shows on par with the previously state-of-the-art without depending on the time consuming autoregressive generation. Further directions for FAME is to (i) test it on larger image datasets with images of a higher resolution, (ii) expand the model to capture other data modalities such as audio and text, (iii) combine the model in a semi-supervised framework.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a generative model that proves state-of-the-art results on gray-scale and natural images.

Represent the TLDR for finding the corresponding scientific article.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising in many sequence learning tasks through sophisticated network designs. This paper shows how we can address this problem in a plain recurrent network by analyzing the gating mechanisms in GNNs. We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates. We compare this model with IRNNs and LSTMs on multiple sequence modeling benchmarks. The RINs demonstrate competitive performance and converge faster in all tasks. Notably, small RIN models produce 12%–67% higher accuracy on the Sequential and Permuted MNIST datasets and reach state-of-the-art performance on the bAbI question answering dataset. Numerous methods have been proposed for mitigating the vanishing gradient problem including the use of second-order optimization methods (e.g., Hessian-free optimization BID15), specific training schedules (e.g., Greedy Layer-wise training BID20 BID7 BID24), and special weight initialization methods when training on both plain FFNs and RNNs BID3 BID16 BID13 BID10 BID26 BID11.Gated Neural Networks (GNNs) also help to mitigate this problem by introducing "gates" to control information flow through the network over layers or sequences. Notable examples include recurrent networks such as Long-short Term Memory (LSTM) BID8, Gated Recurrent Unit (GRU) BID1, and feedforward networks such as Highway Networks (HNs) BID21, and Residual Networks (ResNets) BID5. One can successfully train very deep models by employing these models, e.g., ResNets can be trained with over 1,000 layers. It has been demonstrated that removing (lesioning) or reordering (re-shuffling) random layers in deep feedforward GNNs does not noticeable affect the performance of the network BID23 Noticeably, one interpretation for this effect as given by BID4 is that the functional blocks in HNs or ResNets engage in an Unrolled Iterative Estimate (UIE) of representations and that layers in this block of HNs or ResNets iteratively refine a single set of representations. In this paper, we investigate if the view of Iterative Estimation (IE) can also be applied towards recurrent GNNs (Section 2.1). We present a formal analysis for GNNs by examining a dual gate design common in LSTM and GRU (Section 2.2). The analysis suggests that the use of gates in GNNs encourages the network to learn an identity mapping which can be beneficial in training deep architectures BID6 BID4.We propose a new formulation of a plain RNN, called a Recurrent Identity Network (RIN), that is encouraged to learn an identity mapping without the use of gates (Section 2). This network uses ReLU as the activation function and contains a set of non-trainable parameters. This simple yet effective method helps the plain recurrent network to overcome the vanishing gradient problem while it is still able to model long-range dependencies. This network is compared against two competing networks, the IRNN and LSTM, on several long sequence modeling tasks including the adding problem (Section 3.1), Sequential and Permuted MNIST classification tasks (Section 3.2), and bAbI question answering tasks (Section 3.3). RINs show faster convergence than IRNNs and LSTMs in the early stage of the training phase and reach competitive performance in all benchmarks. Note that the use of ReLU in RNNs usually leads to training instability, and therefore the network is sensitive to training hyperparameters. Our proposed RIN network demonstrates that a plain RNN does not suffer from this problem even with the use of ReLUs as shown in Section 3. We discuss further implications of this network and related work in Section 4. Representation learning in RNNs requires that the network build a latent state, which reflects the temporal dependencies over a sequence of inputs. In this section, we explore an interpretation of this process using iterative estimation (IE), a view that is similar to the UIE view for feedforward GNNs. Formally, we characterize this viewpoint in Eq. 1, that is, the expectation of the difference between the hidden activation at step t, h t, and the last hidden activation at step T, h T, is zero: DISPLAYFORM0 This formulation implies that an RNN layer maintains and updates the same set of representations over the input sequence. Given the fact that the hidden activation at every step is an estimation of the final activation, we derive Eq. 3. Average Estimation Error DISPLAYFORM1 DISPLAYFORM2 Figure 1: Observation of learning identity mapping in an LSTM model trained on the adding problem task (see Section 3.1). The average estimation error is computed over a batch of 128 samples of the test set. (a) and (b) show the evaluation of Eq. 1 and Eq. 3 respectively. The x-axis indicates the index of the step that compares with the final output h T or its previous step h t−1. Fig. 1 shows an empirical observation of the IE in the adding problem (experimental details in Section 3.1). Here, we use the Average Estimation Error (AEE) measure BID4 to quantify the expectation of the difference between two hidden activations. The measured AEEs in Fig. 1 are close to 0 indicating that the LSTM model fulfills the view of IE. The also suggest that the network learns an identity mapping since the activation levels are similar on average across all recurrent updates. In the next section, we shall show that the use of gates in GNNs encourages the network to learn an identity mapping and whether this analysis can be extended to plain recurrent networks. Popular GNNs such as LSTM, GRU; and recent variants such as the Phased-LSTM BID17, and Intersection RNN BID2, share the same dual gate design following: DISPLAYFORM0 where t ∈ [1, T], H t = σ(x t, h t−1) represents the hidden transformation, T t = τ (x t, h t−1) is the transform gate, and C t = φ(x t, h t−1) is the carry gate. σ, τ and φ are recurrent layers that have their trainable parameters and activation functions. represents element-wise product operator. Note that h t may not be the output activation at the recurrent step t. For example in LSTM, h t represents the memory cell state. Typically, the elements of transform gate T t,k and carry gate C t,k are between 0 (close) and 1 (open), the value indicates the openness of the gate at the kth neuron. Hence, a plain recurrent network is a subcase of Eq. 4 when T t = 1 and C t = 0.Note that conventionally, the initial hidden activation h 0 is 0 to represent a "void state" at the start of computation. For h 0 to fit into Eq. 4's framework, we define an auxiliary state h −1 as the previous state of h 0, and T 0 = 1, C 0 = 0. We also define another auxiliary state h T +1 = h T, T T +1 = 0, and C T +1 = 1 as the succeeding state of h T.Based on the recursive definition in Eq. 4, we can write the final layer output h T as follows: DISPLAYFORM1 where we use to represent element-wise multiplication over a series of terms. According to Eq. 3, and supposing that Eq. 5 fulfills the Eq. 1, we can use a zero-mean residual t for describing the difference between the outputs of recurrent steps: DISPLAYFORM2 Plugging Eq. 6 into Eq. 5, we get DISPLAYFORM3 The complete deduction of Eqs. 8-9 is presented in Appendix A. Eq. 8 performs an identity mapping when the carry gate C t is always open. In Eq. 9, the term t i=1 i represents "a level of representation that is formed between h 1 and h t ". Moreover, the term T j=t C j extract the "useful" part of this representation and contribute to the final representation of the recurrent layer. Here, we interpret "useful" as any quantity that helps in minimizing the cost function. Therefore, the contribution, λ t, at each recurrent step, quantifies the representation that is learned in the step t. Furthermore, it is generally believed that a GNN manages and maintains the latent state through the carry gate, such as the forget gate in LSTM. If the carry gate is closed, then it is impossible for the old state to be preserved while undergoing recurrent updates. However, if we set C t = 0, t ∈ [1, T] in Eq. 9, we get: DISPLAYFORM4 If h 0 = 0 (void state at the start), we can turn Eq. 10 into: DISPLAYFORM5 Eq. 11 shows that the state can be preserved without the help of the carry gate. This indicates that it is possible for a plain recurrent network to learn an identity mapping as well. Motivated by the previous iterative estimation interpretation of RNNs, we formulate a novel plain recurrent network variant -Recurrent Identity Network (RIN): DISPLAYFORM0 where W is the input-to-hidden weight matrix, U is the hidden-to-hidden weight matrix, and I is a non-trainable identity matrix that acts as a "surrogate memory" component. This formulation encourages the network to preserve a copy of the last state by embedding I into the hidden-tohidden weights. This "surrogate memory" component maintains the representation encoded in the past recurrent steps. In this section, we compare the performances of the RIN, IRNN, and LSTM in a set of tasks that require modeling long-range dependencies. The adding problem is a standard task for examining the capability of RNNs for modeling longrange dependencies BID8. In this task, two numbers are randomly selected from a long sequence. The network has to predict the sum of these two numbers. The task becomes challenging as the length of the sequence T increases because the relevant numbers can be far from each other in a long sequence. We report experimental from three datasets that have sequence lengths of T 1 = 200, T 2 = 300, and T 3 = 400 respectively. Each dataset has 100,000 training samples and 10,000 testing samples. Each sequence of a dataset has T i numbers that are randomly sampled from a uniform distribution in. Each sequence is accompanied by a mask that indicates the two chosen random positions. We compare the performance between RINs, IRNNs, and LSTMs using the same experimental settings. Each network has one hidden layer with 100 hidden units. Note that a LSTM has four times more parameters than corresponding RIN and IRNN models. The optimizer minimizes the Mean Squared Error (MSE) between the target sum and the predicted sum. We initially used the RMSprop BID22 optimizer. However, some IRNN models failed to converge using this optimizer. Therefore, we chose the Adam optimizer so a fair comparison can be made between the different networks. The batch size is 32. Gradient clipping value for all models is 100. The models are trained with maximum 300 epochs until they converged. The initial learning rates are different between the datasets because we found that IRNNs are sensitive to the initial learning rate as the sequence length increases. The learning rates α 200 = 10 −4, α 300 = 10 DISPLAYFORM0 and α 400 = 10 −6 are applied to T 1, T 2 and T 3 correspondingly. The input-to-hidden weights of RINs and IRNNs and hidden-to-hidden weights of RINs are initialized using a similar method to BID13 where the weights are drawn from a Gaussian distribution N (0, 10 −3). The LSTM is initialized with the settings where the input-to-hidden weights use Glorot Uniform BID3 and hidden-to-hidden weights use an orthogonal matrix as suggested by BID19. Bias values for all networks are initialized to 0. No explicit regularization is employed. We do not perform an exhaustive hyperparameter search in these experiments. The baseline MSE of the task is 0.167. This score is achieved by predicting the sum of two numbers as 1 regardless of the input sequence. FIG1 shows MSE plots for different test datasets. RINs and IRNNs reached the same level of performance in all experiments, and LSTMs performed the worst. Notably, LSTM fails to converge in the dataset with T 3 = 400. The use of ReLU in RINs and IRNNs causes some degree of instability in the training phase. However, in most cases, RINs converge faster and are more stable than IRNNs (see training loss plots in Fig. 5 of Appendix B). Note that because IRNNs are sensitive to the initial learning rate, applying high learning rates such as α = 10 −3 for T 2 and T 3 could cause the training of the network to fail. Sequential and Permuted MNIST are introduced by Le et al. FORMULA0 for evaluating RNNs. Sequential MNIST presents each pixel of the MNIST handwritten image BID14 to the network sequentially (e.g., from the top left corner of the image to the bottom right corner of the image). After the network has seen all 28 × 28 = 784 pixels, the network produces the class of the image. This task requires the network to model a very long sequence that has 784 steps. Permuted MNIST is an even harder task than the Sequential MNIST in that a fixed random index permutation is applied to all images. This random permutation breaks the association between adjacent pixels. The network is expected to find the hidden relations between pixels so that it can correctly classify the image. All networks are trained with the RMSprop optimizer BID22 ) and a batch size of 128. The networks are trained with maximum 500 epochs until they are converged. The initial learning rate is set to α = 10 −6. Weight initialization follows the same setup as Section 3.1. No explicit regularization is added. TAB0 summarizes the accuracy performance of the networks on the Sequential and Permuted MNIST datasets. For small network sizes, RINs outperform IRNNs in their accuracy performance. For bigger networks, RINs and IRNNs achieve similar performance; however, RINs converge much faster than IRNNs in the early stage of training (see FIG2). LSTMs perform the worst on both tasks in terms of both convergence speed and final accuracy. Appendix C presents the full experimental . To investigate the limit of RINs, we adopted the concept of Deep Transition (DT) Networks BID18 for increasing the implicit network depth. In this extended RIN model called RIN-DT, each recurrent step performs two hidden transitions instead of one (the formulation is given in Appendix D). The network modification increases the inherent depth by a factor of two. The showed that the error signal could survive 784 × 2 = 1568 computation steps in RIN-DTs. In FIG3, we show the evidence of learning identity mapping empirically by collecting the hidden activation from all recurrent steps and evaluating Eqs. 1 and 3. The network matches the IE when AEE is close to zero. We also compute the variance of the difference between two recurrent steps. FIG3 suggests that all networks bound the variance across recurrent steps. FIG3 (b) offers a closer perspective where it measures the AEE between two adjacent steps. The levels of activations for all networks are always kept the same on an average, which is an evidence of learning identity mapping. We also observed that the magnitude of the variance becomes significantly larger at the last 200 steps in IRNN and RIN. Repeated application of ReLU may cause this effect during recurrent update BID9. Other experiments in this section exhibit similar behaviors, complete are shown in Appendix C FIG1 ). Note that this empirical analysis only demonstrates that the tested RNNs have the evidence of learning identity mapping across recurrent updates as RINs and IRNNs largely fulfill the view of IE. We do not over-explain the relationship between this analysis and the performance of the network.. The x-axis indicates the index of the step that compares with the final output h T or its previous step h t−1, and y-axis represents the average estimation error (AEE). DISPLAYFORM0 The bAbI dataset provides 20 question answering tasks that measure the understanding of language and the performance of reasoning in neural networks BID25. Each task consists of 1,000 training samples and 1,000 test samples. A sample consists of three parts: a list of statements, a question and an answer (examples in TAB1). The answer to the question can be inferred from the statements that are logically organized together. The red square is below the blue square. Then she journeyed to the garden. The red square is to the left of the pink rectangle. Question: Is the blue square below the pink rectangle? Answer: Garden. Answer: No. We compare the performance of the RIN, IRNN, and LSTM on these tasks. All networks follow a network design where the network firstly embeds each word into a vector of 200 dimensions. The statements are then appended together to a single sequence and encoded by a recurrent layer while another recurrent layer encodes the question sequence. The outputs of these two recurrent layers are concatenated together, and this concatenated sequence is then passed to a different recurrent layer for decoding the answer. Finally, the network predicts the answer via a softmax layer. The recurrent layers in all networks have 100 hidden units. This network design roughly follows the architecture presented in BID11. The initial learning rates are set to α = 10 −3 for RINs and LSTMs and α = 10 −4 for IRNNs because IRNNs fail to converge with a higher learning rate on many tasks. We chose the Adam optimizer over the RMSprop optimizer because of the same reasons as in the adding problem. The batch size is 32. Each network is trained for maximum 100 epochs until the network converges. The recurrent layers in the network follow the same initialization steps as in Section 3.1.The in TAB2 show that RINs can reach mean performance similar to the state-of-theart performance reported in BID11. As discussed in Section 3.1, the use of ReLU as the activation function can lead to instability during training of IRNN for tasks that have lengthy statements (e.g.. 3-Three Supporting Facts, 5-Three Arg. Relations). In this paper, we discussed the iterative representation refinement in RNNs and how this viewpoint could help in learning identity mapping. Under this observation, we demonstrated that the contribution of each recurrent step a GNN can be jointly determined by the representation that is formed up to the current step, and the openness of the carry gate in later recurrent updates. Note in Eq. 9, the element-wise multiplication of C t s selects the encoded representation that could arrive at the output of the layer. Thus, it is possible to embed a special function in C t s so that they are sensitive to certain pattern of interests. For example, in Phased LSTM, the time gate is inherently interested in temporal frequency selection BID17.Motivated by the analysis presented in Section 2, we propose a novel plain recurrent network variant, the Recurrent Identity Network (RIN), that can model long-range dependencies without the use of gates. Compared to the conventional formulation of plain RNNs, the formulation of RINs only adds a set of non-trainable weights to represent a "surrogate memory" component so that the learned representation can be maintained across two recurrent steps. Experimental in Section 3 show that RINs are competitive against other network models such as IRNNs and LSTMs. Particularly, small RINs produce 12%-67% higher accuracy in the Sequential and Permuted MNIST. Furthermore, RINs demonstrated much faster convergence speed in early phase of training, which is a desirable advantage for platforms with limited computing resources. RINs work well without advanced methods of weight initializations and are relatively insensitive to hyperparameters such as learning rate, batch size, and selection of optimizer. This property can be very helpful when the time available for choosing hyperparameters is limited. Note that we do not claim that RINs outperform LSTMs in general because LSTMs may achieve comparable performance with finely-tuned hyperparameters. The use of ReLU in RNNs might be counterintuitive at first sight because the repeated application of this activation is more likely causing gradient explosion than conventional choices of activation function, such as hyperbolic tangent (tanh) function or sigmoid function. Although the proposed IRNN BID13 reduces the problem by the identity initialization, in our experiments, we usually found that IRNN is more sensitive to training parameters and more unstable than RINs and LSTMs. On the contrary, feedforward models that use ReLU usually produce better and converge faster than FFNs that use the tanh or sigmoid activation function. In this paper, we provide a promising method of using ReLU in RNNs so that the network is less sensitive to the training conditions. The experimental also support the argument that the use of ReLU significantly speeds up the convergence. During the development of this paper, a recent independent work BID27 presented a similar network formulation with a focus on training of deep plain FFNs without skip connections. DiracNet uses the idea of ResNets where it assumes that the identity initialization can replace the role of the skip-connection in ResNets. DiracNet employed a particular kind of activation function -negative concatenated ReLU (NCReLU), and this activation function allows the layer output to approximate the layer input when the expectation of the weights are close to zero. In this paper, we showed that an RNN can be trained without the use of gates or special activation functions, which complements the findings and provides theoretical basis in BID27.We hope to see more empirical and theoretical insights that explains the effectiveness of the RIN by simply embedding a non-trainable identity matrix. In future, we will investigate the reasons for the faster convergence speed of the RIN during training. Furthermore, we will investigate why RIN can be trained stably with the repeated application of ReLU and why it is less sensitive to training parameters than the two other models. A ALGEBRA OF EQS. 8-9Popular GNNs such as LSTM, GRU; and recent variants such as the Phased-LSTM BID17, and Intersection RNN BID2, share the same dual gate design described as follows: DISPLAYFORM0 where t ∈ [1, T], H t = σ(x t, h t−1) represents the hidden transformation, T t = τ (x t, h t−1) is the transform gate, and C t = φ(x t, h t−1) is the carry gate. σ, τ and φ are recurrent layers that have their trainable parameters and activation functions. represents element-wise product operator. Note that h t may not be the output activation at the recurrent step t. For example in LSTM, h t represents the memory cell state. Typically, the elements of transform gate T t,k and carry gate C t,k are between 0 (close) and 1 (open), the value indicates the openness of the gate at the kth neuron. Hence, a plain recurrent network is a subcase of Eq. 14 when T t = 1 and C t = 0.Note that conventionally, the initial hidden activation h 0 is 0 to represent a "void state" at the start of computation. For h 0 to fit into Eq. 4's framework, we define an auxiliary state h −1 as the previous state of h 0, and T 0 = 1, C 0 = 0. We also define another auxiliary state h T +1 = h T, T T +1 = 0, and C T +1 = 1 as the succeeding state of h T.Based on the recursive definition in Eq. 4, we can write the final layer output h T as follows: DISPLAYFORM1 where we use to represent element-wise multiplication over a series of terms. According to Eq. 3, and supposing that Eq. 5 fulfills the Eq. 1, we can use a zero-mean residual t for describing the difference between the outputs of recurrent steps: DISPLAYFORM2 Then we can rewrite Eq. 16 as: DISPLAYFORM3 Substituting Eq. 18 into Eq. 15: DISPLAYFORM4 We can rearrange Eqn. 20 to DISPLAYFORM5 The term λ in Eq. 23 can be reorganized to, DISPLAYFORM6 B DETAILS IN THE ADDING PROBLEM EXPERIMENTS Average Estimation Error RIN 2-100 1st IRNN 2-100 1st LSTM 2-100 1st 0 100 200 300 400 500 600 700 800 layer 2 step index RIN 2-100 2nd IRNN 2-100 2nd LSTM 2-100 2nd DISPLAYFORM7 DISPLAYFORM8 In Section 3.2, we tested an additional model for RINs, which takes the concept of Deep Transition Networks (DTNs) BID18. Instead of stacking the recurrent layers, DTNs add multiple nonlinear transitions in a single recurrent step. This modification massively increases the depth of the network. In our RIN-DTs, the number of transition per recurrent step is two. Because the length of the sequence for Sequential and Permuted MNIST tasks is 784, RIN-DTs have the depth of 784 × 2 = 1568. The recurrent layer is defined in Eqs. 26-27. DISPLAYFORM0 DISPLAYFORM1

We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.