Question: represent the text to find a very short summary (1-2 sentences max)

Question: In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the use of a \textit{backtracking model} that predicts the preceding states that terminate at a given high-reward state. We can train a model which, starting from a high value state (or one that is estimated to have high value), predicts and samples which (state, action)-tuples may have led to that high value state. These traces of (state, action) pairs, which we refer to as Recall Traces, sampled from this backtracking model starting from a high value state, are informative as they terminate in good states, and hence we can use these traces to improve a policy. We provide a variational interpretation for this idea and a practical algorithm in which the backtracking model samples from an approximate posterior distribution over trajectories which lead to large rewards. Our method improves the sample efficiency of both on- and off-policy RL algorithms across several environments and tasks. Training control algorithms efficiently from interactions with the environment is a central issue in reinforcement learning (RL). Model-free RL methods, combined with deep neural networks, have achieved impressive across a wide range of domains BID18 BID39. However, existing model-free solutions lack sample efficiency, meaning that they require extensive interaction with the environment to achieve these levels of performance. Model-based methods in RL can mitigate this issue. These approaches learn an unsupervised model of the underlying dynamics of the environment, which does not necessarily require rewards, as the model observes and predicts state-to-state transitions. With a well-trained model, the algorithm can then simulate the environment and look ahead to future events to establish better value estimates, without requiring expensive interactions with the environment. Model-based methods can thus be more sample efficient than their model-free counterparts, but often do not achieve the same asymptotic performance BID6 BID28.In this work, we propose a method that takes advantage of unsupervised observations of state-to-state transitions for increasing the sample efficiency of current model-free RL algorithms, as measured by the number of interactions with the environment required to learn a successful policy. Our idea stems from a simple observation: given a world model, finding a path between a starting state and a goal state can be done either forward from the start or backward from the goal. Here, we explore an idea for leveraging the latter approach and combining it with model-free algorithms. This idea is particularly useful when rewards are sparse. High-value states are rare and trajectories leading to them are particularly useful for a learner. The availability of an exact backward dynamics model of the environment is a strong and often unrealistic requirement for most domains. Therefore, we propose learning a backward dynamics model, which we refer to as a backtracking model, from the experiences performed by the agent. This backtracking model p(s t, a t |s t+1), is trained to predict, given a state s t+1, which state s t the agent visited before s t+1 and what action a t ∼ π was performed in s t to reach s t+1. Specifically, this is a model which, starting from a future high-value state, can be used to recall traces that have ended at this high value state, that is sequences of (state, action)-tuples. This allows the agent to simulate and be exposed to alternative possible paths to reach a high value state. A final state may be a previously experienced high-value state or a goal state may be explicitly given, or even produced by the agent using a generative model of high-value states BID19. Our hypothesis is that using a backtracking model in this way should benefit learning, especially in the context of weak or sparse rewards. Indeed, in environments or tasks where the agent receives rewards infrequently, it must leverage this information effectively and efficiently. Exploration methods have been employed successfully BID2 BID19 BID32 to increase the frequency at which novel states are discovered. Our proposal can be viewed as a special kind of simulated exploration proceeding backward from presumed high-value states, in order to discover trajectories that may lead to high rewards. A backtracking model aims to augment the experience of the trajectory τ leading to a high-value state by generating other possible tracesτ that could have also caused it. To summarize: the main contribution of this paper is an RL method based on the use of a backtracking model, which can easily be integrated with existing on-and off-policy techniques for reducing sample complexity. Empirically, we show with experiments on eight RL environments that the proposed approach is more sample efficient. We consider a Markov decision process (MDP) defined by the tuple (S, A, P, r, γ), where the state space S and the action space A may be discrete or continuous. The learner is not explicitly given the environment transition probability p(s t+1 |s t, a t) for going from s t ∈ S to s t+1 ∈ S given a t ∈ A, but samples from this distribution are observed. The environment emits a bounded reward r: S × A → [r min, r max] on each transition and γ ∈ is the discount factor. Let π denote a stochastic policy over actions given states, and let R(π) = E π T t=0 γ t r(s t) denote the expected total return when policy π is followed. The standard objective in reinforcement learning is to maximize the discounted total return R(π). Throughout the text we will refer to experienced trajectories as τ = (s 1, a 1, · · ·, s T, a T) and we will refer to simulated experiences as tracesτ. We introduce the backtracking model B φ = q φ (s t, a t |s t+1), which is a density estimator of the joint probability distribution over the previous (s t, a t)-tuple parameterized by φ. This distribution is produced by both a learned backward policy π b = q(a t |s t+1) and a state generator q(s t |a t, s t+1). The backward policy predicts the previous action a t given the ing state s t+1. The state generator estimates the probability of a previous state s t given the tuple (a t, s t+1). With these models, we may decompose q φ (s t, a t |s t+1) as q(s t |a t, s t+1)q(a t |s t+1).However, for training stability with continuous-valued states, we model the density of state variation ∆s t = s t − s t+1 rather than the raw s t. Therefore, our density models are given by q φ (∆ t, a t |s t+1) = q(∆s t |a t, s t+1)q(a t |s t+1).Note that for readability, we will drop the φ-subscript unless it is necessary for clarity. Generating Recall Traces. Analogous to the use of forward models in BID31 BID4 ), we may generate a recall trace auto-regressively. To do so, we begin with a state s t+1 and sample a t ∼ q(a t |s t+1). The state generator can then be sampled to produce the change in state ∆s t ∼ q(∆s t |a t, s t+1). We can continue to unroll this process, repeating with state s t = ∆s t + s t+1 for a desired number of steps. These generated transitions are then stored as a potential traceτ which terminates at some final state. The backtracking model B φ is learned by maximum likelihood, using the policy's trajectories as observations, as described in Section 3.1.Producing Intended High Value States. Before recursively sampling from the backtracking model, we need to obtain presumed high-value states. Generally, such states will not be known in advance. However, as the agent learns, it will visit states s t with increasingly high value DISPLAYFORM0. The agent's full experience is maintained in a replay buffer B, in the form of tuples of (s t, a t, s t+1, r t). Filtering of trajectories based on the returns is done, so that only top k traj are added to the buffer. In this work, we will investigate our approach with two methods for generating the initial high-value states. The first method relies on picking the most valuable states stored in the replay buffer B. As before, a valuable state may be defined by its estimated expected return V π (s) as computed by a critic (our off-policy method) or state that received a high reward (our on-policy method).The second method is based on Goal GAN, recently introduced by BID19 where goal states g are produced via a Generative Adversarial Network BID10. In our variant, we map the goal state g to a valid point in state space s using a'decoder' D. For the point-mass, the goal and state are identical; for Ant, we use a valid random joint-angle configuration at that goal position. The backtracking model is then used to find plausible trajectories that terminate at that state. For both methods, as the learner improves, one would expect that on average, higher value states are used to seed the recall traces. In this section, we describe how to train the backtracking model and how it can be used to improve the efficiency of the agent's policy and aid with exploration. We use a maximum likelihood training loss for training of the backtracking model B φ on the top k% of the agent's trajectories stored in the state buffer B. At each iteration, we perform stochastic gradient updates based on agent trajectories τ, with respect to the following objective: DISPLAYFORM0 logq(a t |s t+1) + logq(∆s t |a t, s t+1), where s t = ∆s t +s t+1 and T is the episode length. For our chosen backtracking model, this implies a mean-squared error loss (i.e. corresponding to a conditional Gaussian for ∆s t) for continuous action tasks and a cross-entropy loss (i.e. corresponding to a conditional Multinoulli for s t given a t and s t+1) for discrete action tasks. The buffer is constantly updated with recent experiences and the backtracking model is trained online in order to encourage generalization as the distribution of trajectories in the buffer evolves. We now describe how we use the recall tracesτ to improve the agent's policy π θ. In brief, the traces τ generated by the backtracking model are used as observations for imitation learning BID33 ros, 2011; BID3 by the agent. The backtracking model will be continuously updated as new actual experiences are generated, as described in Section 3.1. Imitation learning of the policy is performed simply by maximizing the log-probability of the agent's action a t given s t given by DISPLAYFORM0 where (s t, a t)-tuples come from a generated traceτ.Our motivation for having the agent imitate trajectories from the backtracking model is two-fold: Execute policy to produce trajectory τ 5: DISPLAYFORM1 Estimate ∇ θ R(π θ) from RL algorithm 7: DISPLAYFORM2 Compute L B via Equation 2, using top k% valuable states from top k traj trajectories in B 9: DISPLAYFORM3 Obtain target high value state s (see Algorithm 2 for details)11:Generate N recall tracesτ for s using B φ (s) Compute imitation loss L I via Equation FORMULA3 13: DISPLAYFORM0 Dealing with sparse rewards States with significant return are emphasized by the backtracking model, since the traces it generates are initialized at high value states. We expect this behaviour to help in the context of sparse or weak rewards. The backtracking model can also generate new ways to reach high-value states. So even if it cannot directly discover new high-value states, it can at least point to new ways to reach known high value states, thus aiding with exploration. Thus far, we have motivated the use of a backtracking model intuitively. In this section, we provide a motivation relying on a variational perspective of RL and ideas from the wake-sleep algorithm BID21.Let R be the return of a policy trajectory τ, i.e. the sum of discounted rewards under this trajectory. Consider the event of the return R being larger than some threshold L. The probability of that event under the agent's policy is p(R > L) = τ p(R > L|τ)p(τ), where p(τ) is distribution of trajectories under policy π, p(R > L|τ) = 1 R>L and 1 A is the indicator function that is equal to 1 if A is true and is otherwise 0.Let q(τ) be any other distribution over trajectories, then we have the following classic relationship between the marginal log-probability of an observation (R > L) and the KL-divergence between q and the posterior over a latent variable (τ): DISPLAYFORM0 where DISPLAYFORM1 This suggests an EM-style training procedure, that alternates between training the variational distribution q(τ) towards the posterior p(τ |R > L) and training the policy to maximize L. In this context, we view the backtracking model and the high-value states sampler as providing q(τ) implicitly. Specifically, we assume that q factorizes temporally as in Equation 2 with the backtracking model providing q(a t |s t+1) and q(∆s t |a t, s s+1). We parameterize the approximate posterior in this way so that we can take advantage of a model of the backwards transitions to conveniently sample from q starting from a high-value final state s T. This makes sense in the context of sparse rewards, where few states have significant reward. If we have a way to identify these high-reward states, then it is much easier to obtain these posterior trajectories by starting from them. Training q(τ) by minimizing the KL(q(τ)||p(τ |R > L)) term is hard due to the direction of the KL-divergence. Taking inspiration from the wake-sleep algorithm BID21, we instead minimize the KL in the opposite direction, KL(p(τ |R > L)||q(τ)). This can be done by sampling trajectories from p(τ |R > L) (e.g. by rejection sampling, keeping only the forward-generated trajectories which lead to R > L) and maximizing their log-probability under q(τ). This recovers our algorithm, which trains the backtracking model on high-return trajectories generated by the agent. So far we have assumed a known threshold L. However, in practice the choice of L is important. While ultimately we would want L to be close to the highest possible return, at the early stages of training it cannot be, as trajectories from the agent are unlikely to reach that threshold. A better strategy is to gradually increase L. One natural way of doing this is to use the top few percentile trajectories sampled by the agent for training q(τ), instead of explicitly setting L. This approach can be thought of as providing a curriculum for training the agent that is adapted to its performance. This is also related to evolutionary methods BID17 BID1, which keep the "fittest" samples from a population in order to re-estimate a model, from which new samples are generated. This variational point of view also tells us how the prior over the last state should be constructed. The ideal prior q(s T) is simply a generative model of the final states leading to R > L. Both methods proposed to estimate q(s T) with this purpose, either non-parametrically (with the forward samples for which R > L) or parametrically (with generative model trained from those samples). Also, if goal states are known ahead of time, then we can set L as the reward of those states (minus a small quantity) and we can seed the backwards trajectories from these goal states. In that case the variational objective used to train the policy is a proxy for log-likelihood of reaching a goal state. Control as inference The idea of treating control problems as inference has been around for many years BID40 BID23 BID44 BID45 BID34. A good example of this idea is to use Expectation Maximization (EM) for RL BID5, of which the PoWER algorithm BID24 is one well-known practical implementation. In EM algorithms for RL, learning is divided between the estimation of the expectation over the trajectories conditioned on the reward observations and estimation of a new policy based on these expectation estimates. While we don't explicitly try to estimate these expectations, one could argue that the samples from the backtracking model serve a similar purpose. Variational inference has also been proposed for policy search BID29 BID26. Probabilistic views of the RL problem have also been used to construct maximum entropy methods for both regular and inverse RL BID15 ).Using off-policy trajectories By incorporating the trajectories of a separate backtracking model, our method is similar in spirit to approaches which combine on-policy learning algorithms with off-policy samples. Recent examples of this, like the interpolated policy gradient BID14, PGQ BID30 and ACER BID46, combine policy gradient learning with ideas for off-policy learning and methodology inspired by Q-learning. Our method differs by using the backtracking model to obtain off-policy trajectories and is, as an idea, independent of the specific model-free RL method it is combined with. Our work to effectively propagate value updates backwards is also related to the seminal work of prioritized sweeping .Model-based methods A wide range of model-based RL and control methods have been proposed in the literature BID8. PILCO (b), is a model-based policy search method to learn a probabilistic model of dynamics and incorporate model uncertainty into long-term planning. The classic Dyna (Sutton) algorithm was proposed to take advantage of a model to generate simulated experiences, which could be included in the training data for a modelfree algorithm. This method was extended to work with deep neural network policies, but performed best with models that were not neural networks BID13. Other extensions to Dyna have also been proposed BID38 BID22 BID18.Other approaches have also been proposed to combine advantages of both value and policy-based approaches BID27 BID41. Finally, BID9 ) is concurrent work to this and also proposes training on imagined reversal steps from known goal states. BID11 proposed to use similar learning rule in the context of generative models to modify the parameters of transition operator to make the reverse of this heated trajectory more likely under a reverse cooling process. Our experimental evaluation aims to understand whether our method can improve the sample complexity of off-policy as well as on-policy RL algorithms. Practically, we must choose the length of generated backward traces. Longer traces become increasingly likely to deviate significantly from the traces that the agent can generate from its initial state. Therefore, in our experiments, we sample fairly short tracesτ from the backtracking model, where the length is adjusted manually based on the time-scale of each task. We empirically show the following across different experimental settings:• Samples from the true backtracking model can be used to improve sample efficiency.• Using a learned backtracking model starting from high value states accelerates learning for off-policy as well as on-policy experiments.• Modeling parametrically and generating high value states (using GoalGAN) also helps. Here, we aim to check if in the ideal case when the true backtracking model is known, the proposed approach works. To investigate this, we use the four-room environment from BID35 of various dimensions. The 4-room grid world is a simple environment where the agent must navigate to a goal position to receive a positive reward through bottleneck states (doorways). We compare the proposed method to the scenario where the policy is trained through the actor-critic method BID25 with generalized advantage estimation (GAE) BID37 Finding the goal state becomes more challenging as the dimension increases due to sparsity of rewards. Therefore, we expect that the backtracking model would be a more effective tool in larger environments. Confirming this hypothesis, we see in FIG1 that, as we increase the dimensionality of the maze, sample efficiency increases thanks to recall traces, compared to our baseline. Here we aim to compare the performance of recall traces with Prioritized Experience Replay (PER). PER stores the past experiences in a buffer and then selectively trains on high value experiences. We again use the Four-room Environment. PER gives an optimistic bias to the critic, and while it allows the reinforcement of sparse rewards, it also converges too quickly to an exploitation mode, which can be difficult to get out of. In order to show this, we plot the state visitation counts of a policies trained with PER and recall traces and see that the latter visit more states. We show in FIG3 that while PER is competitive in the smaller 11x11 environment, recall traces outperform it in the larger 15x15 environment. FIG2 also shows how the use of a backtracking model and recall traces pushes the policy to visit a wider variety of grid positions than PER. One situation in which states to start the backtracking model at are naturally available, is when the method is combined with an algorithm for sub-goal selection. We chose to investigate how well the backtracking model can be used in conjunction with the automatic goal generation algorithm from BID19, Goal GAN. It uses a Generative Adversarial Network to produce sub-goals at the appropriate level of difficulty for the agent to reach. As the agent learns, new sub-goals of increasing difficulty are generated. This way, the agent is pressured to explore and learn to be able to reach any location in the state space. We hypothesize that the backtracking model should help the agent to reach the sub-goals faster and explore more efficiently. Hence, in this learning scenario, what changes is that high value states are now generated by Goal GAN instead of being selected by a critic from a replay buffer. We performed experiments on the U-Maze Ant task described in BID19. It is a challenging robotic locomotion task where a quadruped robot has to navigate its center of mass within some particular distance of a target goal. The objective is to cover as much of the space of the U-shaped maze as possible. We find that using the backtracking model improves data efficiency, by reaching a coverage of more than 63% in 155 steps, instead of 275 steps without it FIG4. More visualizations and learning curves for U-Maze Ant task as well as the N-Dimensional Point Mass task can be found in Appendix FIG0 and 13). We conducted robotic locomotion experiments using the MuJoCo simulator BID43. We use the same setup as BID28. We compare our approach with a pure model-free method on standard benchmark locomotion tasks, to learn the fastest forward-moving gait possible. The model-free approach we consider is the rllab implementation of trust region policy optimization (TRPO) BID36. For the TRPO baseline we use the same setup as BID28. See the appendix for the model implementation details. The in Figure 6 show that our method consistently outperforms TRPO on all of the benchmark tasks in terms of final performance, and learns substantially faster. Figure 6: Our model as compared to TRPO. For TRPO baselines, except walker, we ran with 5 different random seeds. For our model, we ran with 5 different random seeds. Here, we evaluate on the same range of challenging continuous control tasks from the OpenAI gym benchmark suite. We compare to Soft Actor Critic (SAC) BID16, shown to be more sample efficient compared to other off-policy algorithms such as DDPG BID18 and which consistently outperforms DDPG. Part of the reason for choosing SAC instead of DDPG was that the latter is also known to be more sensitive to hyper-parameter settings, limiting its effectiveness on complex tasks BID20. For SAC, we use the same hyper-parameters reported in BID16. Implementation details for our model are listed in the Appendix. Performance across tasks -The in Figure 7 show that our method consistently improves the performance of SAC on all of the benchmark tasks, leading to faster learning. In fact, the largest improvement is observed on the hardest task, Ant. Figure 7: Our model as compared to SAC. We ran SAC baselines with 2 different random seeds. For our model, we ran with 5 different random seeds. We advocate for the use of a backtracking model for improving sample efficiency and exploration in RL. The method can easily be combined with popular RL algorithms like TRPO and soft actorcritic. Our indicate that the recall traces generated by such models are able to accelerate learning on a variety of tasks. We also show that the method can be combined with automatic goal generation. The Appendix provides more analysis on the sensitivity of our method to various factors and ablations. We show that a random model is outperformed by a trained backtracking model, confirming its usefulness and present plots showing the effect of varying the length of recall traces. For future work, while we observed empirically that the method has practical value and could relate its workings from a variational perspective, but more could be done to improve our theoretical understanding of its convergence behavior and what kind of assumptions need to hold about the environment. It would also be interesting to investigate how the backtracking model can be combined with forward models from a more conventional model-based system. Return argmax(V (s)) ∀s ∈ B 7: end if In FIG7 we show the performance in learning efficiency when the length of the backward traces is varied. The backtracking model we used for all the experiments consisted of two multi-layer perceptrons: one for the backward action predictor Q(a t |s t+1) and one for the backward state predictor Q(s t |a t, s t+1). Both MLPs had two hidden layers of 128 units. The action predictor used hyperbolic tangent units while the inverse state predictor used ReLU units. Each network produced as output the mean and variance parameters of a Gaussian distribution. For the action predictor the output variance was fixed to 1. For the state predictor this value was learned for each dimension. We do about a hundred training-steps of the backtracking model for every 5 training-steps of the RL algorithm. For training the backtracking model we maintain a buffer which stores states (state, action, nextstate, reward) yielding high rewards. We sample a batch from the buffer and then normalize the states, actions before training the backtracking model on them. During the sampling phase we feed in the normalized nextstate in the backward action predictor Q(a t |s t+1) to get normalized action. Then we input this normalized action in the backward state predictor Q(s t |a t, s t+1) to get normalized previous state. We then un-normalize the obtained previous states and action using the corresponding mean and variance to compute the Imitation loss. This is required for stability during sampling. FOUR ROOM ENVIRONMENT An agent has to navigate within − distance of the goal position at the end of a U-shaped maze. For the Point Mass U-Maze navigation, we use high return trajectories for training the parameters of backtracking model (i.e those trajectories which reach sub-goals defined by Goal GAN). We sample a trajectory of length 20 from our backtracking model and the recall traces are used to improve the policy via imitation learning. For the Ant U-Maze navigation, we train identically, only with length-50 traces. FIG0. It is clear from the figure that it is helpful to learn the backward action predictor Q(a t |s t+1). FIG0 shows the same comparison for Ant-v1, and Walker2d-v1 where the baseline is Soft Actor Critic (SAC). For our baseline i.e the scenario when backtracking model is not learned, we experimented with all the lengths 1,2,3,4,5, 10 and we choose the best one as our baseline. As you can see in the FIG0, backtracking model performs best as compared to SAC baseline as well as to the scenario when backtracking model is not trained. Hence proving that backtracking model is not just doing random search. Dyna algorithm uses a forward model to generate simulated experience that could be included in a model-free algorithm. This method can be used to work with deep neural network policies, but performed best with models which are not neural networks BID12. Our intuition says that it might be better to generate simulated experience from backtracking model (starting from a high value state) as compared to forward model, just because we know that traces from backtracking model are good, as they lead to high value state, which is not really the case for the simulated experience from a forward model. FIG0 we evaluate the Forward model with On-Policy TRPO on Ant and Humanoid Mujoco tasks. We were not able to get any better on with forward model as compared to the Baseline TRPO, which is consistent with the findings from BID12. Building the backward model is necessarily neither harder nor easier. Realistically, building any kind of model and having it be accurate for more than, say, 10 time steps is pretty hard. But if we only have 10 time steps of accurate transitions, it is probably better to take them backward model from different states as compared to from forward model from the same initial state. (as corroborated by our experiments).Something which remains as a part of future investigation is to train the forward model and backtracking model jointly. As backtracking model is tied to high value state, the forward model could extract the goal value from the high value state. When trained jointly, this should help the forward model learn some reduced representation of the state that is necessary to evaluate the reward. Ultimately, when planning, we want the model to predict the goal accurately, which helps to optimize for this "goal-oriented" behaviour directly. This also avoids the need to model irrelevant aspects of the environment. Here, we show the of various ablations in the four-room environment which highlight the effect various hyperparameters have on performance. In FIG0 we train on the recall traces after a fixed number of iterations of learning in the true environment. For all of the environments, as we increase the ratio of updates in the true environment to updates using recall traces from backward model, the performance decreases significantly. This again highlights the advantages of learning from recall traces. In FIG0, we see the effects of training from the recall traces multiple times for every iteration of training in the true environment. We can see that as we increase the number of iteration of learning from recall traces, we correspondingly need to choose smaller trace length. For each update in the real environment, making more number of updates helps if the trace length is smaller, and if the trace length is larger, it has a detrimental effect on the learning process as is seen in 18. Also as observed in the cases of 12x12 and 14x14 environment, it may happen that for an increased ratio of learning from recall traces and high trace length, the model achieves the maximum reward initially but after sometime the average reward plummets.: Legend indicates the ratio of updates in the true environment to updates using recall traces. Here we learn more in the true environment than using traces. We can see that not learning regularly from recall traces gives decreased performance. FIG0: Legend indicates the ratio of updates in the true environment to updates using recall traces with the trajectory length of traces in parenthesis. Here we learn more using recall traces than actual environment. This tells that more updates using recall traces helps if we have a correspondingly lower trajectory length We investigate the effect of doing more updates from the generated recall traces on some Mujoco tasks using Off-Policy SAC. As can be seen from 19, we find that using more traces helps and that for an increased number of updates we need to correspondingly shorten the trajectory length of the sampled traces. These experiments show that there is a balance between how much we should train in the actual environment and how much we should learn from the traces generated from the backward model. In the smaller four room-environment, 1:1 balance performed the best. In Mujoco tasks and larger four room environments, doing more updates from the backward model helps, but in the smaller four room maze, doing more updates is detrimental. So depending upon the complexity of the task, we need to decide this ratio.

Represent the short summary for finding the corresponding scientific article (long)

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We augment the Q-value estimates with a count-based bonus that ensures optimism during action selection and bootstrapping, even if the Q-value estimates are pessimistic.


Question: Represent text

Question: Prepositions are among the most frequent words. Good prepositional representation is of great syntactic and semantic interest in computational linguistics. Existing methods on preposition representation either treat prepositions as content words (e.g., word2vec and GloVe) or depend heavily on external linguistic resources including syntactic parsing, training task and dataset-specific representations. In this paper we use word-triple counts (one of the words is a preposition) to capture the preposition's interaction with its head and children. Prepositional embeddings are derived via tensor decompositions on a large unlabeled corpus. We reveal a new geometry involving Hadamard products and empirically demonstrate its utility in paraphrasing of phrasal verbs. Furthermore, our prepositional embeddings are used as simple features to two challenging downstream tasks: preposition selection and prepositional attachment disambiguation. We achieve comparable to or better than state of the art on multiple standardized datasets. Prepositions are a linguistically closed class comprising some of the most frequent words; they play an important role in the English language since they encode rich syntactic and semantic information. Many preposition-related tasks still remain unsolved in computational linguistics because of their polysemous nature and flexible usage patterns. An accurate understanding and representation of prepositions' linguistic role is key to several important NLP tasks such as grammatical error correction and prepositional phrase attachment. A first-order approach is to represent prepositions as real-valued vectors via word embeddings such as word2vec BID21 and GloVe BID25.Word embeddings have brought a renaissance in NLP research; they have been very successful in capturing word similarities as well as analogies (both syntactic and semantic) and are now mainstream in nearly all downstream NLP tasks (such as question-answering). Despite this success, no specific properties of word embeddings of prepositions have been highlighted in the literature. Indeed, many of the common prepositions have very similar vector representations as shown in TAB0 for preposition vectors trained using word2vec and GloVe. While this suggests that using available representations for prepositions diminishes the distinguishing feature between prepositions, one could hypothesize that this is primarily because standard word embedding algorithms treat prepositions no different from other content words such as verbs and nouns, i.e., embeddings are created based on co-occurrences with other words. However, prepositions are very frequent and co-occur with nearly all words, which means that their co-occurrence ought to be treated differently. Modern descriptive linguistic theory proposes to understand a preposition via its interactions with both the head (attachment) and child (complement) BID12; BID8. This theory naturally suggests that one should count co-occurrences of a given preposition with pairs of neighboring words. One way of achieving this would be by considering a tensor of triples (word 1, word 2, preposition), where we do not restrict word 1 and word 2 to be head-and child-words; instead we model a preposition's interaction with all pairs of neighboring words via a slice of a tensor X -the slice is populated by word co-occurrences restricted to a context window of the specific preposition. Thus, the tensor dimension is V × V × K where V is the vocabulary and K is the number of prepositions; since K ≈ 50, we note that V K.Using such a representation, we find that the ing tensor is low rank and extract embeddings for both preposition and non-preposition words using a combination of standard ideas from word representations (such as weighted spectral decomposition as in GloVe BID25) and tensor decompositions (alternating least squares (ALS) methods BID29 ). The preposition embeddings are discriminative, see preposition similarity of the tensor embedding in TAB0. We demonstrate that the ing representation for prepositions captures the core linguistic property of prepositions. We do this using both intrinsic evaluations and downstream tasks, where we provide new state-of-the-art on well-known NLP tasks involving prepositions. Intrinsic evaluations: We show that the Hadamard product of the embeddings of a verb and a preposition closely approximates the representation of this phrasal verb's paraphrase. Example: v made v from ≈ v produced where represents the Hadamard product (i.e., pointwise multiplication) of two vectors; this approximation does not hold for the standard word embeddings of prepositions (word2vec or GloVe). We provide a mathematical interpretation for this new geometry as well as empirically demonstrate the generalization on a new data set of compositional phrasal verbs. Extrinsic evaluations: Our preposition embeddings are used as features for a simple classifier in two well-known challenging downstream NLP classification tasks. In both tasks, we perform comparable to or strictly better than the state-of-the-art on multiple standardized datasets. Preposition selection: The choice of prepositions significantly influences (and is governed by) the semantics of the context they occur in. Furthermore, the prepositional choice is usually very subtle (and consequently is one of the most frequent error types made by second language English speakers BID19). This task tests the choice of a preposition in a large set of contexts (7, 000 instances of both CoNLL-2013 and SE datasets BID26). Our approach achieves 6% and 2% absolute improvement over the previous state-of-the-art on the respective datasets. Prepositional attachment disambiguation: Prepositional phrase attachment is a common cause of structural ambiguity in natural language. In the sentence "Pierre Vinken joined the board as a voting member", the prepositional phrase "as a voting member" can attach to either "joined" (the VP) or "the board" (the NP); in this case the VP attachment is correct. Despite extensive study over decades of research, prepositional attachment continues to be a major source of syntactic parsing errors BID4;; BID7. We use our prepositional representations as simple features to a standard classifier on this task. Our approach tested on a widely studied standard dataset BID2 achieves 89% accuracy, essentially the same performance as state-of-the-art (90% accuracy). It is noteworthy that while the stateof-the-art are obtained with significant linguistic resources, including syntactic parsers and WordNet, our approach does not rely on these resources to achieve a comparable performance. We emphasize two aspects of our contributions: It is folklore within the NLP community that representations via pairwise word counts capture much of the benefits of the unlabeled sentence-data; example: BID29 reports that their word representations via word-triple counts are better than others, but still significantly worse than regular word2vec representations. One of our main observations is that considering word-triple counts makes most (linguistic) sense when one of the words is a preposition. Furthermore, the sparsity of the corresponding tensor is no worse than the sparsity of the regular word co-occurrence matrix (since prepositions are so frequent and co-occur with essentially every word). Taken together, these two points strongly suggest the benefits of tensor representations in the context of prepositions. The word and preposition representations via tensor decomposition are simple features leading to a standard classifier. In particular, we do not use syntactic parsing (which many prior methods have relied on) or handcrafted features BID26 or train task-specific representations on the annotated training dataset BID2. The simplicity combined with our strong empirical (new state-of-the-art on long standing datasets) lends credence to the strength of the prepositional representations found via tensor decompositions. We begin with a description of how the tensor with triples (word,word,preposition) is formed and empirically show that its slices are low-rank. Next, we derive low dimensional vector representations for words and prepositions via appropriate tensor decomposition methods. Tensor creation: Suppose that K prepositions are in the preposition set P = {p 1, . . ., p K}; here K is 49 in our preposition selection task, and 76 in the attachment disambiguation task. The vocabulary, the set of all words except prepositions, contains N words V = {w 1, . . ., w N}, and N ≈ 1M. We generate a third order tensor X N ×N ×(K+1) from WikiCorpus Al- BID0 in the following way. We say two words co-occur if they appear within distance t of each other in a sentence. For k ≤ K, the entry X ijk is the number of occurrences where word w i co-occurs with preposition p k, and w j also co-occurs with preposition p k in the same sentence, and this is counted across all sentences in a large WikiCorpus. Here we use a window of size t = 3. There are also a number of words which do not occur in the context of any preposition. To make full use of the data, we add an extra slice X[:, :, K + 1]: the entry X ij(K+1) is the number of occurrences where w i co-occurs with w j (within distance 2t = 6) but at least one of them is not within a distance of t of any preposition. Note that the preposition window of 3 is smaller than the word window of 6, since it is known that the interaction between prepositions and neighboring words usually weakens more sharply with the distance as compared to content words BID11.Empirical properties of X: We find that the tensor X is very sparse -only 1% of tensor elements are non-zeros. Furthermore, every slice log(1 + X[:, :, k]) is low rank (here the logarithm is applied componentwise to every entry of the tensor slice). We choose slices corresponding to prepositions "about", "before","for", "in" and "of", and plot their normalized singular values in FIG0. We see that the singular values decay dramatically, suggesting the low rank structure in each slice. Tensor decomposition: We combine standard ideas from word embedding algorithms and tensor decomposition algorithms to arrive at the low rank approximation to the tensor log(1 + X). In particular, we consider two separate methods: A generic method to decompose the tensor into its modes is via the CANDECOMP/PARAFAC (CP) decomposition BID15. The tensor log(1 + X) is decomposed into three modes: U d×N, W d×N and Q d×(K+1), based on the solutions to the optimization problem. Here u i, w i and q i are the i-th column of U, W and Q, respectively. DISPLAYFORM0 where a, b, c = 1 t (a b c) is the inner product of three vectors a, b and c. Here 1 is the column vector of all ones and refers to the Hadamard product. We can interpret the columns of U as the word representations and the columns of Q as the preposition representations, each of dimension d (equal to 200 in this paper). There are several algorithmic solutions to this optimization problem in the literature, most of which are based on alternating least squares methods BID15 BID5 BID1 and we employ a recent one named Orth-ALS BID29 in this paper. Orth-ALS periodically orthogonalizes decomposed components while fixing two modes and updating the remaining one. It is supported by theoretical guarantees and empirically outperforms standard ALS method in different applications. DISPLAYFORM1 where b U i is the scalar bias for word i in matrix U. Similarly, b W j is the bias for word j in matrix W, and b Qk is for preposition k in matrix Q. Bias terms are learned to minimize the loss function. Here ω ijk is the weight assigned to each tensor element X ijk, and we use the weighting proposed byGloVe: ω ijk = min X ijk xmax α, 1. We set hyperparameters x max = 10, and α = 0.75 in this work. We solve this optimization problem via standard gradient descent, arriving at word representations U and tensor representations Q. Representation Interpretation Suppose that we have a phrase (h, p i, c) where h, p i and c are head word, preposition i(1 ≤ i ≤ K) and child respectively. A phrase example is split off something. The inner product of word vectors of h, p i and c reflects how frequently h and c cooccur in the context of p. It also reflects how cohesive the triple is. Recall that there is an extra (K + 1)−th slice that describes the word co-occurrences outside the preposition window, which considers cases such as the phrasal verb (v, c) where v and c are the verb and the child. The verb phrase divide something is equivalent to the phrase split off something. For any word c that fits in this phrase semantically, we can expect that DISPLAYFORM0 In other words u h q i ≈ u v q K+1, where a b denotes the pointwise multiplication (Hadamard product) of vectors a and b. This suggests that we could paraphrase the verb phrase (h, p i) by finding the verb v such that DISPLAYFORM1 Well-trained embeddings should be able to capture the relation between the prepositional phrases and their equivalent phrasal verbs. In TAB1, we list seven paraphrases of verb phrases, as generated from the weighted tensor decomposition. A detailed list of paraphrases on a new dataset of compositional verb phrases is available in TAB0 in Appendix B, where we also compare paraphrasing using regular word embeddings and via both addition and Hadamard product operations. The combination of tensor representations and Hadamard product in vastly superior paraphrasing. In the next two sections, we evaluate tensor-based preposition embeddings in the context of two important NLP downstream tasks: preposition selection and preposition attachment disambiguation. In this work, we use WikiCorpus as the training corpus for different sets of embeddings. We train tensor embeddings with both Orth-ALS and weighted decomposition. The implementation of Orth-ALS is built upon he SPLATT toolkit BID31. We perform orthogonalization in the first 5 iterations in Orth-ALS decomposition, and the training is completed when its performance stabilizes. As for the weighted decomposition, we train for 20 iterations, and its hyperparameters are set as x max = 10, and α = 0.75.We also include two baselines, word2vec's CBOW model and GloVe, for comparison. We set 20 training iterations to both models. Hyperparameters in word2vec are set as: window size=6, negative sampling=25 and down sampling=1e-4. Hyperparameters in GloVe are set as: window size=6, x max =10, α=0.75 and minimum word count=5. We note that all the representations in this studyword2vec, GloVe and our tensor embedding -are of dimension 200. The detection and correction of grammatical errors is an important task in NLP. Second language learners tend to make more mistakes and in particular, prepositional errors make up about 13% of all errors, ranking the second among most common error types BID19. This is due to the fact that prepositions are highly polysemous and have flexible usage. Accurate preposition selection needs to well capture the interaction between preposition and its context. This task is natural to evaluate how well the lexical interactions are captured by different methods. Task. Given a sentence in English with a preposition, we either replace the preposition (to the correct one) or retain it. For example, "to" should be corrected as "of" in the sentence "It can save the effort to carrying a lot of cards". Formally, there is a closed set of preposition candidates P = {p 1, . . ., p m}. A preposition p is used in a sentence s consisting of words s = {. . ., w −2, w −1, p, w 1, w 2, . . .}. If used incorrectly, we need to replace p by another prepositionp ∈ P based on the context. TAB2. We focus on the most frequent 49 prepositions listed in Appendix A.Evaluation metric. Three metrics, precision, recall and F1 score (harmonic mean of precision and recall) are used to evaluate preposition selection performance. Our algorithm. We first preprocess the dataset by removing articles, determiners and pronouns, and take a context window of 3. We divide the task into two steps: error identification and error correction. Firstly, we decide whether a preposition is used correctly in the context. If not, we suggest another preposition as replacement in the second step. Identification step uses only three features: cosine similarity between the current preposition embedding and the average context embedding, rank of the preposition in terms of cosine similarity, and probability that this preposition is not changed in training corpus. We build a decision tree classifier with these three features and find that we can identify errors with 98% F1 score in the CoNLL dataset and 96% in the SE dataset. When it comes to error correction, we only focus on identified errors in the first stage. Suppose that the original preposition is q, and the candidate preposition is p.; Confusion probability: the probability that q is replaced by p in the training data. A two-layer feedforward neural network (FNN) with hidden sizes of 500 and 10 is trained with these features to score prepositions in each sentence. The one with the highest score is the suggested edit. Baseline. State-of-the-art on preposition selection uses n-gram statistics from a large corpus BID26. Features such as pointwise mutual information (PMI) and part-of-speech tags are fed into a supervised scoring system. Prepositions with highest score are chosen as suggested ones. The performance is affected by both the system architecture and features. To evaluate the benefits brought by our tensor embedding-based features, we also consider other baselines which have the same two-step architecture whereas features are generated from word2vec and GloVe embeddings. These baselines allow us to compare the representation power independent of the classifier. Result. We compare our methods against baselines mentioned above in TAB3. As is seen, tensor embeddings achieve the best performance among all approaches. In particular, tensor with weighted decomposition has the highest F1 score on CoNLL dataset, 6% improvement over the state of the art. The tensor with ALS decomposition performs the best on SE dataset, achieving 2% improvement. We also note that with the same architecture, tensor embeddings perform much better than word2vec and GloVe embeddings on both datasets. It validates the representation power of tensor embeddings. To have a deeper insight into feature importance in the preposition selection task, we also perform an ablation analysis of the tensor method with weighted decomposition as shown in TAB4. We remove one feature each time, and report the performance achieved by remaining features. It is found that left context is the most important feature in CoNLL dataset, whereas confusion score is the most important in SE dataset. Pair similarity and triple similarity are less important compared with other features. This is because the neural network could learn lexical similarity from embedding features, and diminishes the importance of similarity features. Discussion. Now we analyze the reasons why our approach selects wrong prepositions in some sentences. Limited context window. We focus on the local context within preposition's window. In some cases, we find that head words might be out of the context window. In the sentence "prevent more of this kind of tragedy to happening" where to should be corrected as from. Given the context window of 3, we cannot get the lexical clues provided by prevent, which leads to the selection error in our approach. Preposition selection requires more context. Even when the context window contains all words on which the preposition depends, it still may not be sufficient to select the right preposition. For example, in the sentence "it is controlled by bad men in a not good purpose" where our approach replaces the preposition in with the preposition on given the high frequency of the phrase "on purpose". The correct preposition should be for based on the whole sentence. In this section, we discuss prepositional phrase (PP) attachment disambiguation, a well-studied, but still open, hard task in syntactic parsing. A prepositional phrase usually consists of head words, a preposition and child words. An example is "he saw an elephant with long tusks", where "with" is attached to the noun "elephant". In another example "he saw an elephant with his telescope", "with" is attached to the verb "saw". Head words can be different when only child word is changed. PP attachment disambiguation inherently requires accurate description of interactions among head, preposition and child, which becomes an ideal task to evaluate our tensor-based embeddings. Task. The English dataset used in our work is collected from a linguistic treebank by BID2. TAB5 enumerates statistics associated with this dataset. Each instance consists of several head candidates, a preposition and a child word. We need to pick the head to which the preposition is attached. In the examples above, words "saw" and "elephant" are head candidates. Our algorithm. Let v h, v p and v c be embeddings for the head candidate h, preposition p and child c respectively. Features we use for the attachment disambiguation are: embedding feature: candidate, preposition and child embedding; triple similarity: triple sim(h, p, c) = DISPLAYFORM0; part-of-speech (pos) tag of the candidate and its next word; (b) distance between h and p. We use a basic neural network, a two-layer feedforward network (FNN) with hidden sizes of 1000 and 20 to take input features and predict the probability that a candidate is the head. The candidate with the highest likelihood is chosen as the head. Baseline. We include following state-of-the-art approaches in preposition attachment disambiguation. The linguistic resources they used to enrich features are listed in Table 7. Head-Prep-Child-Dist (HPCD) Model BID2: this compositional neural network is used to train task-specific word representations. Low-Rank Feature Representation (LRFR) BID33: this method incorporates word parts, contexts and labels into a tensor, and uses decomposed vectors as features for disambiguation. Ontology LSTM (OntoLSTM) BID6: Word vectors are initialized with GloVeextended from AutoExtend BID27, and then trained via LSTMs for head selection. Similar to the experiments in preposition selection, we also include baselines which have the same feedforward network architecture but generate features with vectors trained by word2vec and GloVe. They are denoted as FNN with different initializations in Table 7. Since the attachment disambiguation is a selection task, accuracy is a natural evaluation metric. Result. We compare and linguistic resources of different approaches in Table 7, where we see that our simple classifier built on the tensor representations is within 1% of the state of the art; prior state of the art have used significant linguistic resources enumerated in Table 7. With the same feedforward neural network as the classifier, our tensor-based approaches (both ALS and WD) achieve better performance than word2vec and GloVe. Ablation analysis in TAB6 shows that head vector feature affects the performance most (indicating that heads interact more closely with prepositions), and POS tag comes second. Similarity features appear less important since the classifier has access to lexical relatedness via the embedding features. Distance feature is reported to be important in previous works since 81.7% sentences take the word closest to the preposition as their head. In our experiments, distance becomes less important compared with embedding features. Discussion. We find that one source of attachment disambiguation error is the lack of broader context in our features. Broader context is critical in examples such as "worked" and "system" which are head candidates of "for trades" in a sentence. They are reasonable heads in expressions "worked for trades" and "system for trades". It requires more context to decide that "system" rather than "worked" is the head in the given sentence. We further explore the difference in identifying head verbs and head nouns. We have found that tensor's geometry could aid in paraphrasing verb phrases, and thus it well captures the interaction between verbs and prepositions. In this task, we want to see whether our approach could do better in identifying head verbs than head nouns. There are 883 instances with head verbs on which we could achieve an accuracy of 0.897, and 1068 instances with head nouns where the accuracy is 0.887. We do better in selecting head verbs, but performance does not differ too much across verbs and nouns. Tensor Decomposition. Tensors embed higher order interaction among different modes, and the tensor decomposition captures the relations via lower dimensional representations. There are several decomposition methods such as Alternating Least Square (ALS) BID15, Simultaneous Diagonalization (SD) BID16 and optimization-based methods BID20 BID23. Orthogonalized Alternating Least Square (Orth-ALS) adds the step of component orthogonalization to each update step in the ALS method BID29. Orth-ALS, supported by theoretical guarantees and, more relevantly, good empirical performance, is the algorithm of choice in this paper. Preposition Selection. Preposition selection, a major area of study in both syntactic and semantic computational linguistics, is also a very practical topic in the context of grammar correction and second language learning. Prior works typically use hand-crafted heuristic rules in preposition correction BID32; lexical n-gram features are also known to be very useful BID26; BID28. Syntactic information such as POS tags and dependency parsing can further enrich features BID13, and are standard in generic tasks involving prepositions. Prepositional Attachment Disambiguation. There is a storied literature on prepositional attachment disambiguation, long recognized as an important part of syntactic parsing BID14. Recent works, based on word embeddings have pushed the boundary of state of the art empirical . A seminal work in this direction is the Head-Prep-Child-Dist (HPCD) Model, which trained word embeddings in a compositional network designed to maximize the accuracy of head prediction BID2. A very recent work has proposed an initialization with semantics-enriched GloVe embeddings, and retrained representations with LSTM-RNNs BID6. Another recent work has used tensor decompositions to capture the relation between word representations and their labels BID33. Co-occurrence counts of word pairs in sentences and the ing word vector representations (embeddings) have revolutionalized NLP research. A natural generalization is to consider co-occurrence counts of word triples, ing in a third order tensor. Partly due to the size of the tensor (a vocabulary of 1M, leads to a tensor with 10 18 entries!) and partly due to the extreme dynamic range of entries (including sparsity), word vector representations via tensor decompositions have largely been inferior to their lower order cousins (i.e., regular word embeddings).In this work, we trek this well-trodden terrain but restricting word triples to the scenario when one of the words is a preposition. This is linguistically justified, since prepositions are understood to model interactions between pairs of words. Numerically, this is also very well justified since the sparsity and dynamic range of the ing tensor is no worse than the original matrix of pairwise co-occurrence counts; this is because prepositions are very frequent and co-occur with essentially every word in the vocabulary. Our intrinsic evaluations and new state of the art in downstream evaluations lend strong credence to the tensor-based approach to prepositional representation. We expect our vector representations of prepositions to be widely used in more complicated downstream NLP tasks where prepositional role is crucial, including "text to programs" BID10. The list of most frequent 49 Prepositions in the task of preposition selection is shown below: about, above, absent, across, after, against, along, alongside, amid, among, amongst, around, at, before, behind, below, beneath, beside, besides, between, beyond, but, by, despite, during, except, for, from, in, inside, into, of, off, on, onto, opposite, outside, over, since, than, through, to, toward, towards, under, underneath, until, upon, with. B PARAPHRASING OF PHRASAL VERBS In Section 3 we have provided a simple linear algebraic method to generate paraphrases to compositional phrasal verbs. We approximate the paraphrase representation u v via Eq. 3, and get a list of words which have similar representations as candidate paraphrases. These candidates do not include words that are the same as component words in the phrase. We also require that a reasonable paraphrase should be a verb. Therefore we choose the verb which is most similar to u v among candidates. We filter verbs with Python NLTK tools BID3 and Linguistics library of.Sample examples of the top paraphrases are provided in TAB1. Here we provide a detailed enumeration of the of our linear algebraic method on a new dataset of 60 compositional phrases. In the paraphrasing task, we consider three sets of embeddings, word2vec, GloVe and tensor embeddings from weighted decomposition. We also have two composition methods: addition and Hadamard product to approximate the paraphrase representation from verb and preposition vectors. Addition is included here because it has been widely used to approximate phrasal embedding in previous works BID22 BID9. We enumerate paraphrases generated by six combinations of embeddings and composition methods, validating the representation power of tensor embeddings and the multiplication (Hadamard product) composition method. As we can see from TAB7 and 10, tensor embedding works better with multiplicative composition, whereas word2vec and GloVe work better with additive composition. Overall, tensor embedding together with multiplication gives better paraphrases than other approaches.

This work is about tensor-based method for preposition representation training.

represent the short summary for finding the corresponding scientific article (long)

For example, 'We extend a state-of-the-art technique to directly incorporate FLOPs as part of the optimization objective, and we show that, given a desired FLOPs requirement, different neural networks are successfully trained.' should be close to 'There exists a plethora of techniques for inducing structured sparsity in parametric models during the optimization process, with the final goal of resource-efficient inference. However, to the best of our knowledge, none target a specific number of floating-point operations (FLOPs) as part of a single end-to-end optimization objective, despite reporting FLOPs as part of the . Furthermore, a one-size-fits-all approach ignores realistic system constraints, which differ significantly between, say, a GPU and a mobile phone -- FLOPs on the former incur less latency than on the latter; thus, it is important for practitioners to be able to specify a target number of FLOPs during model compression. In this work, we extend a state-of-the-art technique to directly incorporate FLOPs as part of the optimization objective and show that, given a desired FLOPs requirement, different neural networks can be successfully trained for image classification. Neural networks are a class of parametric models that achieve the state of the art across a broad range of tasks, but their heavy computational requirements hinder practical deployment on resourceconstrained devices, such as mobile phones, Internet-of-things (IoT) devices, and offline embedded systems. Many recent works focus on alleviating these computational burdens, mainly falling under two non-mutually exclusive categories: manually designing resource-efficient models, and automatically compressing popular architectures. In the latter, increasingly sophisticated techniques have emerged BID3 BID4 BID5, which have achieved respectable accuracy-efficiency operating points, some even Pareto-better than that of the original network; for example, network slimming BID3 reaches an error rate of 6.20% on CIFAR-10 using VGGNet BID9 with a 51% FLOPs reduction-an error decrease of 0.14% over the original. However, to the best of our knowledge, none of the methods impose a FLOPs constraint as part of a single end-to-end optimization objective. MorphNets BID0 apply an L 1 norm, shrinkage-based relaxation of a FLOPs objective, but for the purpose of searching and training multiple models to find good network architectures; in this work, we learn a sparse neural network in a single training run. Other papers directly target device-specific metrics, such as energy usage BID15, but the pruning procedure does not explicitly include the metrics of interest as part of the optimization objective, instead using them as heuristics. Falling short of continuously deploying a model candidate and measuring actual inference time, as in time-consuming neural architectural search BID11, we believe that the number of FLOPs is reasonable to use as a proxy measure for actual latency and energy usage; across variants of the same architecture, Tang et al. suggest that the number of FLOPs is a stronger predictor of energy usage and latency than the number of parameters BID12.Indeed, there are compelling reasons to optimize for the number of FLOPs as part of the training objective: First, it would permit FLOPs-guided compression in a more principled manner. Second, practitioners can directly specify a desired target of FLOPs, which is important in deployment. Thus, our main contribution is to present a novel extension of the prior state of the art BID6 to incorporate the number of FLOPs as part of the optimization objective, furthermore allowing practitioners to set and meet a desired compression target. Formally, we define the FLOPs objective L f lops: f × R m → N 0 as follows: DISPLAYFORM0 where L f lops is the FLOPs associated with hypothesis h(·; θ θ θ):= p(·|θ θ θ), g(·) is a function with the explicit dependencies, and I is the indicator function. We assume L f lops to depend only on whether parameters are non-zero, such as the number of neurons in a neural network. For a dataset D, our empirical risk thus becomes DISPLAYFORM1 Hyperparameters λ f ∈ R + 0 and T ∈ N 0 control the strength of the FLOPs objective and the target, respectively. The second term is a black-box function, whose combinatorial nature prevents gradient-based optimization; thus, using the same procedure in prior art BID6, we relax the objective to a surrogate of the evidence lower bound with a fully-factorized spike-and-slab posterior as the variational distribution, where the addition of the clipped FLOPs objective can be interpreted as a sparsity-inducing prior p(θ θ θ) DISPLAYFORM2 where denotes the Hadamard product. To allow for efficient reparameterization and exact zeros, Louizos et al. BID6 propose to use a hard concrete distribution as the approximation, which is a stretched and clipped version of the binary Concrete distribution BID7: ifẑ ∼ BinaryConcrete(α, β), thenz:= max(0, min(1, (ζ − γ)ẑ + γ)) is said to be a hard concrete r.v., given ζ > 1 and γ < 0. Define φ φ φ:= (α α α, β), and let ψ(φ φ φ) = Sigmoid(log α α α − β log DISPLAYFORM3 ψ(·) is the probability of a gate being non-zero under the hard concrete distribution. It is more efficient in the second expectation to sample from the equivalent Bernoulli parameterization compared to hard concrete, which is more computationally expensive to sample multiple times. The first term now allows for efficient optimization via the reparameterization trick BID2; for the second, we apply the score function estimator (REINFORCE) BID14, since the FLOPs objective is, in general, nondifferentiable and thus precludes the reparameterization trick. High variance is a non-issue because the number of FLOPs is fast to compute, hence letting many samples to be drawn. At inference time, the deterministic estimator isθ θ θ:= θ θ θ max(0, min(1, Sigmoid(log α α α)(ζ − γ) + γ)) for the final parametersθ θ θ. In practice, computational savings are achieved only if the model is sparse across "regular" groups of parameters, e.g., each filter in a convolutional layer. Thus, each computational group uses one hard concrete r.v. BID6 -in fully-connected layers, one per input neuron; in 2D convolution layers, one per output filter. Under convention in the literature where one addition and one multiplication each count as a FLOP, the FLOPs for a 2D convolution layer h conv (·; θ θ θ) given a random draw z is then defined as L f lops (h conv, z) = (K w K h C in + 1)(I w − K w + P w + 1)(I h − K h + P h + 1) z 0 for kernel width and height (K w, K h), input width and height (I w, I h), padding width and height (P w, P h), and number of input channels C in. The number of FLOPs for a fully-connected layer h f c (·; θ θ θ) is L f lops (h f c, z) = (I n + 1) z 0, where I n is the number of input neurons. Note that these are conventional definitions in neural network compression papers-the objective can easily use instead a number of FLOPs incurred by other device-specific algorithms. Thus, at each training step, we compute the FLOPs objective by sampling from the Bernoulli r.v.'s and using the aforementioned definitions, e.g., L f lops (h conv, ·) for convolution layers. Then, we apply the score function estimator to the FLOPs objective as a black-box estimator. We report on MNIST, CIFAR-10, and CIFAR-100, training multiple models on each dataset corresponding to different FLOPs targets. We follow the same initialization and hyperparameters as Louizos et al. BID6, using Adam BID1 with temporal averaging for optimization, a weight decay of 5 × 10 −4, and an initial α that corresponds to the original dropout rate of that layer. We similarly choose β = 2/3, γ = −0.1, and ζ = 1.1. For brevity, we direct the interested reader to their repository BID0 for specifics. In all of our experiments, we replace the original L 0 penalty with our FLOPs objective, and we train all models to 200 epochs; at epoch 190, we prune the network by weights associated with zeroed gates and replace the r.v.'s with their deterministic estimators, then finetune for 10 more epochs. For the score function estimator, we draw 1000 samples at each optimization step-this procedure is fast and has no visible effect on training time. BID10 7-13-208-16 1.1% 254K SBP BID8 3-18-284-283 0.9% 217K BC-GNJ BID5 8-13-88-13 1.0% 290K BC-GHS BID5 5-10-76-16 1.0% 158K L 0 BID6 20-25-45-462 0.9% 1.3M L 0 -sep BID6 9-18-65-25 1.0% 403K DISPLAYFORM0 We choose λ f = 10 −6 in all of the experiments for LeNet-5-Caffe, the Caffe variant of LeNet-5. BID0 We observe that our methods TAB0, bottom three rows) achieve accuracy comparable to those from previous approaches while using fewer FLOPs, with the added benefit of providing a tunable "knob" for adjusting the FLOPs. Note that the convolution layers are the most aggressively compressed, since they are responsible for most of the FLOPs in this model. Orig. in TAB1 denotes the original WRN-28-10 model BID16, and L 0 -* refers to the L 0 -regularized models BID6; likewise, we augment CIFAR-10 and CIFAR-100 with standard random cropping and horizontal flipping. For each of our (last two rows), we report the median error rate of five different runs, executing a total of 20 runs across two models for each of the two datasets; we use λ f = 3 × 10 −9 in all of these experiments. We also report both the expected FLOPs and actual FLOPs, the former denoting the number of FLOPs, on average, at training time under stochastic gates and the latter denoting the number of FLOPs at inference time. We restrict the FLOPs calculations to the penalized non-residual convolution layers only. For CIFAR-10, our approaches in Pareto-better models with decreases in both error rate and the actual number of inference-time FLOPs. For CIFAR-100, we do not achieve a Pareto-better model, since our approach trades accuracy for improved efficiency. The acceptability of the tradeoff depends on the end application.'

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network. Likelihood-based training of these models requires restricting their architectures to allow cheap computation of Jacobian determinants. Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation. In this paper, we use Hutchinson’s trace estimator to give a scalable unbiased estimate of the log-density. The is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures. We demonstrate our approach on high-dimensional density estimation, image generation, and variational inference, achieving the state-of-the-art among exact likelihood methods with efficient sampling. Reversible generative models use cheaply invertible neural networks to transform samples from a fixed base distribution. Examples include NICE BID3, Real NVP BID4, and Glow BID12. These models are easy to sample from, and can be trained by maximum likelihood using the change of variables formula. However, this requires placing awkward restrictions on their architectures, such as partitioning dimensions or using rank one weight matrices, in order to avoid an O(D 3) cost determinant computation. In contrast to directly parameterizing a normalized distribution (e.g. BID17 ; BID5), the change of variables formula allows one to specify a complex normalized distribution p x (x) implicitly by warping a normalized base distribution p z (z) through an invertible function f: R D → R D. Given a random variable z ∼ p z (z) the log density of x = f (z) follows log p x (x) = log p z (z) − log det ∂f (z) ∂zwhere ∂f (z) /∂z is the Jacobian of f. In general, computing the log determinant has a time cost of O(D 3). Much work has gone into developing restricted neural network architectures which make computing the Jacobian's determinant more tractable. These approaches broadly fall into three categories:Normalizing flows. By restricting the functional form of f, various determinant identities can be exploited . These models cannot be trained as generative models from data because they do not have a tractable inverse f −1. However, they are useful for specifying approximate posteriors for variational inference BID13. Autoregressive transformations. By using an autoregressive model and specifying an ordering of the dimensions, the Jacobian of f is enforced to be lower triangular BID14 BID16. These models excel at density estimation for tabular datasets , but require D sequential evaluations of f to invert, which is prohibitive when D is large. Partitioned transformations. Partitioning the dimensions and using affine transformations makes the determinant of the Jacobian cheap to compute, and the inverse f −1 computable with the same cost as f BID3 BID4. This method allows the use of convolutional architectures, excelling at density estimation for image data BID4 BID12.Throughout this work, we refer to reversible generative models as those which use the change of variables to transform a base distribution to the model distribution while maintaining both efficient density estimation and efficient sampling capabilities using a single pass of the model. There exist several approaches to generative modeling approaches which do not use the change of variables equation for training. Generative adversarial networks (GANs) BID6 use large, unrestricted neural networks to transform samples from a fixed base distribution. Lacking a closed-form likelihood, an auxiliary discriminator model must be trained to estimate divergences or density ratios in order to provide a training signal. Autoregressive models BID5 BID17 directly specify the joint distribution p(x) as a sequence of explicit conditional distributions using the product rule. These models require at least O(D) evaluations to sample from. Variational autoencoders (VAEs) BID13 use an unrestricted architecture to explicitly specify the conditional likelihood p(x|z), but can only efficiently provide a stochastic lower bound on the marginal likelihood p(x). define a generative model for data x ∈ R D similar to those based on, but replace the warping function with an integral of continuous-time dynamics. The generative process first samples from a base distribution z 0 ∼ p z0 (z 0). Then, given an ODE whose dynamics are defined by the parametric function ∂z(t) /∂t = f (z(t), t; θ), we solve the initial value problem with z(t 0) = z 0 to obtain a data sample x = z(t 1). These models are called Continous Normalizing Flows (CNF). The change in log-density under this model follows a second differential equation, called the instantaneous change of variables formula BID2: DISPLAYFORM0 We can compute total change in log-density by integrating across time: Given a datapoint x, we can compute both the point z 0 which generates x, as well as log p(x) under the model by solving the combined initial value problem: DISPLAYFORM1 DISPLAYFORM2 which integrates the combined dynamics of z(t) and the log-density of the sample backwards in time from t 1 to t 0. We can then compute log p(x) using the solution of and adding log p z0 (z 0). The existence and uniqueness of require that f and its first derivatives be Lipschitz continuous BID10, which can be satisfied in practice using neural networks with smooth Lipschitz activations, such as softplus or tanh. CNFs are trained to maximize. This objective involves the solution to an initial value problem with dynamics parameterized by θ. For any scalar loss function which operates on the solution to an initial value problem DISPLAYFORM0 then shows that its derivative takes the form of another initial value problem DISPLAYFORM1 The quantity − ∂L /∂z(t) is known as the adjoint state of the ODE. BID2 use a black-box ODE solver to compute z(t 1), and then a separate call to a solver to compute with the initial value ∂L /∂z(t1). This approach is a continuous-time analog to the backpropgation algorithm (; BID1 and can be combined with gradient-based optimization to fit the parameters θ by maximum likelihood. Switching from discrete-time dynamics to continuous-time dynamics reduces the primary computational bottleneck of normalizing flows from DISPLAYFORM0, at the cost of introducing a numerical ODE solver. This allows the use of more expressive architectures. For example, each layer of the original normalizing flows model of is a one-layer neural network with only a single hidden unit. In contrast, the instantaneous transformation used in planar continuous normalizing flows BID2) is a one-layer neural network with many hidden units. In this section, we construct an unbiased estimate of the log-density with O(D) cost, allowing completely unrestricted neural network architectures to be used. In general, computing Tr (∂f /∂z(t)) exactly costs O(D 2), or approximately the same cost as D evaluations of f, since each entry of the diagonal of the Jacobian requires computing a separate derivative of f BID7. However, there are two tricks that can help. First, vector-Jacobian products v T ∂f ∂z can be computed for approximately the same cost as evaluating f using reverse-mode automatic differentiation. Second, we can get an unbiased estimate of the trace of a matrix by taking a double product of that matrix with a noise vector: DISPLAYFORM0 The above equation holds for any D-by-D matrix A and distribution p over D-dimensional vectors such that E[] = 0 and Cov = I. The Monte Carlo estimator derived from FORMULA7 is known as Hutchinson's trace estimator BID9 BID0.To keep the dynamics deterministic within each call to the ODE solver, we can use a fixed noise vector for the duration of each solve without introducing bias: DISPLAYFORM1 Typical choices of p are a standard Gaussian or Rademacher distribution BID9. Often, there exist bottlenecks in the architecture of the dynamics network, i.e. hidden layers whose width H is smaller than the dimensions of the input D. In such cases, we can reduce the variance of Hutchinson's estimator by using the cyclic property of trace. Since the variance of the estimator for Tr(A) grows asymptotic to ||A|| DISPLAYFORM0 When f has multiple hidden layers, we choose H to be the smallest dimension. This bottleneck trick can reduce the norm of the matrix which may also help reduce the variance of the trace estimator. As introducing a bottleneck limits our model capacity, we do not use this trick in our experiments. However this trick can reduce variance when a bottleneck is used, as shown in our ablation studies. Our complete method uses the dynamics defined in and the efficient log-likelihood estimator of to produce the first scalable and reversible generative model with an unconstrained Jacobian. We call this method Free-Form Jacobian of Reversible Dyanamics (FFJORD). Pseudo-code of our method is given in Algorithm 1, and TAB1 summarizes the capabilities of our model compared to other recent generative modeling approaches. Assuming the cost of evaluating f is on the order of O(DH) where D is the dimensionality of the data and H is the size of the largest hidden layer in f, then the cost of computing the likelihood in models with repeated use of invertible transformations FORMULA0 is DISPLAYFORM0 where L is the number of transformations used. For CNF, this reduces to O((DH + D 2)L) for CNFs, whereL is the number of evaluations of f used by the ODE solver. With FFJORD, this reduces further to DISPLAYFORM1 Algorithm 1 Unbiased stochastic log-density estimation using the FFJORD model Require: dynamics f θ, start time t 0, stop time t 1, data samples x, data dimension D.← sample unit variance(x.shape) Sample outside of the integral DISPLAYFORM2 Augment f with log-density dynamics. DISPLAYFORM3 Compute vector-Jacobian product with automatic differentiation DISPLAYFORM4 Figure 2: Comparison of trained Glow, planar CNF, and FFJORD models on 2-dimensional distributions, including multi-modal and discontinuous densities. We demonstrate FFJORD on a variety of density estimation tasks, and for approximate inference in variational autoencoders BID13. Experiments were conducted using a suite of GPU-based ODE-solvers and an implementation of the adjoint method for backpropagation 1. In all experiments the RungeKutta 4 algorithm with the tableau from was used to solve the ODEs. We ensure tolerance is set low enough so numerical error is negligible; see Appendix C.We used Hutchinson's trace estimator during training and the exact trace when reporting test . This was done in all experiments except for our density estimation models trained on MNIST and CIFAR10 where computing the exact Jacobian trace was too expensive. The dynamics of FFJORD are defined by a neural network f which takes as input the current state z(t) ∈ R D and the current time t ∈ R. We experimented with several ways to incorporate t as an input to f, such as hyper-networks, but found that simply concatenating t on to z(t) at the input to every layer worked well and was used in all of our experiments. We first train on 2 dimensional data to visualize the model and the learned dynamics.2 In FIG1, we show that by warping a simple isotropic Gaussian, FFJORD can fit both multi-modal and even discontinuous distributions. The number of evaluations of the ODE solver is roughly 70-100 on all datasets, so we compare against a Glow model with 100 discrete layers. The learned distributions of both FFJORD and Glow can be seen in FIG1. Interestingly, we find that Glow learns to stretch the unimodal base distribution into multiple modes but has trouble modeling the areas of low probability between disconnected regions. In contrast, FFJORD is capable of modeling disconnected modes and can also learn convincing approximations of discontinuous density functions (middle row in FIG1). Since the main benefit of FFJORD is the ability to train with deeper dynamics networks, we also compare against planar CNF BID2 BID8, cannot be sampled from without resorting to correlated or expensive sampling algorithms such as MCMC.On MNIST we find that FFJORD can model the data as effectively as Glow and Real NVP using only a single flow defined by a single neural network. This is in contrast to Glow and Real NVP which must compose many flows to achieve similar performance. When we use multiple flows in a multiscale architecture (like those used by Glow and Real NVP) we obtain better performance on MNIST and comparable performance to Glow on CIFAR10. Notably, FFJORD is able to achieve this performance while using less than 2% as many parameters as Glow. We also note that Glow uses a learned base distribution whereas FFJORD and Real NVP use a fixed Gaussian. A summary of our on density estimation can be found in TAB4 and samples can be seen in Figure 3. Full details on architectures used, our experimental procedure, and additional samples can be found in Appendix B.1.In general, our approach is slower than competing methods, but we find the memory-efficiency of the adjoint method allows us to use much larger batch sizes than those methods. On the tabular datasets we used a batch sizes up to 10,000 and on the image datasets we used a batch size of 900. We compare FFJORD to other normalizing flows for use in variational inference. In VAEs it is common for the encoder network to also output the parameters of the flow as a function of the input x. With FFJORD, we found this led to differential equations which were too difficult to integrate numerically. Instead, the encoder network outputs a low-rank update to a global weight matrix and an input-dependent bias vector. When used in recognition nets, neural network layers defining the dynamics inside FFJORD take the form DISPLAYFORM0 where h is the input to the layer, σ is an element-wise activation function, D in and D out are the input and output dimension of this layer, andÛ (x),V (x),b(x) are input-dependent parameters returned from an encoder network. A full description of the model architectures used and our experimental setup can be found in Appendix B.2.On every dataset tested, FFJORD outperforms all other competing normalizing flows. A summary of our variational inference can be found in TAB6. We performed a series of ablation experiments to gain a better understanding of the proposed model. We plotted the training losses on MNIST using an encoder-decoder architecture (see Appendix B.1 for details). Loss during training is plotted in FIG2, where we use the trace estimator directly on the D×D Jacobian, or we use the bottleneck trick to reduce the dimension to H × H. Interestingly, we find that while the bottleneck trick can lead to faster convergence when the trace is estimated using a Gaussian-distributed, we did not observe faster convergence when using a Rademacherdistributed. The full computational cost of integrating the instantaneous change of variables FORMULA1 is O(DH L) where D is dimensionality of the data, H is the size of the hidden state, and L is the number of function evaluations (NFE) that the adaptive solver uses to integrate the ODE. In general, each evaluation of the model is O(DH) and in practice, H is typically chosen to be close to D. Since the general form of the discrete change of variables equation FORMULA0 Figure 5: NFE used by the adaptive ODE solver is approximately independent of data-dimension. Lines are smoothed using a Gaussian filter. We train VAEs using FFJORD flows with increasing latent dimension D. The NFE throughout training is shown in Figure 5. In all models, we find that the NFE increases throughout training, but converges to the same value, independent of D. We conjecture that the number of evaluations is not dependent on the dimensionality of the data but the complexity of its distribution, or more specifically, how difficult it is to transform its density into the base distribution. 200 400 600 800 NFE 0.9 DISPLAYFORM0 Bits/dimSingle FFJORD Multiscale FFJORD Figure 6: For image data, a single FFJORD flow can achieve near performance to multi-scale architecture while using half the number of evaluations. Crucial to the scalability of Real NVP and Glow is the multiscale architecture originally proposed in BID4. We compare a single-scale encoder-decoder style FFJORD with a multiscale FFJORD on the MNIST dataset where both models have a comparable number of parameters and plot the total NFE-in both forward and backward passes-against the loss achieved in Figure 6. We find that while the single-scale model uses approximately one half as many function evaluations as the multiscale model, it is not able to achieve the same performance as the multiscale model. Number of function evaluations can be prohibitive. The number of function evaluations required to integrate the dynamics is not fixed ahead of time, and is a function of the data, model architecture, and model parameters. This number tends to grow as the models trains and can become prohibitively large, even when memory stays constant due to the adjoint method. Various forms of regularization such as weight decay and spectral normalization BID15 can be used to reduce the this quantity, but their use tends to hurt performance slightly. Limitations of general-purpose ODE solvers. In theory, our model can approximate any differential equation (given mild assumptions based on existence and uniqueness of the solution), but in practice our reliance on general-purpose ODE solvers restricts us to non-stiff differential equations that can be efficiently solved. ODE solvers for stiff dynamics exist, but they evaluate f many more times to achieve the same error. We find that a small amount of weight decay regularizes the ODE to be sufficiently non-stiff. We have presented FFJORD, a reversible generative model for high-dimensional data which can compute exact log-likelihoods and can be sampled from efficiently. Our model uses continuoustime dynamics to produce a generative model which is parameterized by an unrestricted neural network. All required quantities for training and sampling can be computed using automatic differentiation, Hutchinson's trace estimator, and black-box ODE solvers. Our model stands in contrast to other methods with similar properties which rely on restricted, hand-engineered neural network architectures. We demonstrated that this additional flexibility allows our approach to achieve on-par or improved performance on density estimation and variational inference. We believe there is much room for further work exploring and improving this method. FFJORD is empirically slower to evaluate than other reversible models like Real NVP or Glow, so we are interested specifically in ways to reduce the number of function evaluations used by the ODE-solver without hurting predictive performance. Advancements like these will be crucial in scaling this method to even higher-dimensional datasets. We thank Yulia Rubanova and Roger Grosse for helpful discussions. Samples from our FFJORD models trained on MNIST and CIFAR10 can be found in Figure 7. Figure 7: Samples and data from our image models. MNIST on left, CIFAR10 on right. On the tabular datasets we performed a grid-search over network architectures. We searched over models with 1, 2, 5, or 10 flows with 1, 2, 3, or 4 hidden layers per flow. Since each dataset has a different number of dimensions, we searched over hidden dimensions equal to 5, 10, or 20 times the data dimension (hidden dimension multiplier in TAB10). We tried both the tanh and softplus nonlinearities. The best performing models can be found in the TAB10.On the image datasets we experimented with two different model architectures; a single flow with an encoder-decoder style architecture and a multiscale architecture composed of multiple flows. While they were able to fit MNIST and obtain competitive performance, the encoder-decoder architectures were unable to fit more complicated image datasets such as CIFAR10 and Street View House Numbers. The architecture for MNIST which obtained the in TAB4 was composed of four convolutional layers with 64 → 64 → 128 → 128 filters and down-sampling with strided convolutions by two every other layer. There are then four transpose-convolutional layers who's filters mirror the first four layers and up-sample by two every other layer. The softplus activation function is used in every layer. The multiscale architectures were inspired by those presented in BID4. We compose multiple flows together interspersed with "squeeze" operations which down-sample the spatial resolution of the images and increase the number of channels. These operations are stacked into a "scale block" which contains N flows, a squeeze, then N flows. For MNIST we use 3 scale blocks and for CIFAR10 we use 4 scale blocks and let N = 2 for both datasets. Each flow is defined by 3 convolutional layers with 64 filters and a kernel size of 3. The softplus nonlinearity is used in all layers. Both models were trained with the Adam optimizer BID11. We trained for 500 epochs with a learning rate of.001 which was decayed to.0001 after 250 epochs. Training took place on six GPUs and completed after approximately five days. Our experimental procedure exactly mirrors that of. We use the same 7-layer encoder and decoder, learning rate (.001), optimizer , batch size, and early stopping procedure (stop after 100 epochs of no validaiton improvment). The only difference was in the nomralizing flow used in the approximate posterior. We performed a grid-search over neural network architectures for the dynamics of FFJORD. We searched over networks with 1 and 2 hidden layers and hidden dimension 512, 1024, and 2048. We used flows with 1, 2, or 5 steps and wight matrix updates of rank 1, 20, and 64. We use the softplus activation function for all datasets except for Caltech Silhouettes where we used tanh. The best performing models can be found in the TAB11. Models were trained on a single GPU and training took between four hours and three days depending on the dataset. Table 6: Negative log-likehood on test data for density estimation models. Means/stdev over 3 runs. Real NVP, MADE, MAF, TAN, and MAF-DDSF on are taken from BID8. In reproducing Glow, we were able to get comparable to the reported Real NVP by removing the invertible fully connected layers. ODE solvers are numerical integration methods so there is error inherent in their outputs. Adaptive solvers (like those used in all of our experiments) attempt to predict the errors that they accrue and modify their step-size to reduce their error below a user set tolerance. It is important to be aware of this error when we use these solvers for density estimation as the solver outputs the density that we report and compare with other methods. When tolerance is too low, we run into machine precision errors. Similarly when tolerance is too high, errors are large, our training objective becomes biased and we can run into divergent training dynamics. Since a valid probability density function integrates to one, we take a model trained on FIG0 and numerically find the area under the curve using Riemann sum and a very fine grid. We do this for a range of tolerance values and show the ing error in FIG3. We set both atol and rtol to the same tolerance. The numerical error follows the same order as the tolerance, as expected. During training, we find that the error becomes non-negligible when using tolerance values higher than 10 −5. For most of our experiments, we set tolerance to 10 −5 as that gives reasonable performance while requiring few number of evaluations. For the tabular experiments, we use atol=10 −8 and rtol=10 −6.

We use continuous time dynamics to define a generative model with exact likelihoods and efficient sampling that is parameterized by unrestricted neural networks.

represent the short summary for finding the corresponding scientific article (long).

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO