Question: Represent the scientific article to find a short TLDR

Question: In reinforcement learning, it is common to let an agent interact with its environment for a fixed amount of time before resetting the environment and repeating the process in a series of episodes. The task that the agent has to learn can either be to maximize its performance over (i) that fixed amount of time, or (ii) an indefinite period where the time limit is only used during training. In this paper, we investigate theoretically how time limits could effectively be handled in each of the two cases. In the first one, we argue that the terminations due to time limits are in fact part of the environment, and propose to include a notion of the remaining time as part of the agent's input. In the second case, the time limits are not part of the environment and are only used to facilitate learning. We argue that such terminations should not be treated as environmental ones and propose a method, specific to value-based algorithms, that incorporates this insight by continuing to bootstrap at the end of each partial episode. To illustrate the significance of our proposals, we perform several experiments on a range of environments from simple few-state transition graphs to complex control tasks, including novel and standard benchmark domains. Our show that the proposed methods improve the performance and stability of existing reinforcement learning algorithms. The reinforcement learning framework BID19 BID2 BID21 BID10 involves a sequential interaction between an agent and its environment. At every time step t, the agent receives a representation S t of the environment's state, selects an action A t that is executed in the environment which in turn provides a representation S t+1 of the successor state and a reward signal R t+1. An individual reward received by the agent does not directly indicate the quality of its latest action as some rewards may indeed be the consequence of a series of actions taken far in advance. Thus, the goal of the agent is to learn a good policy by maximizing the discounted sum of future rewards also known as return: DISPLAYFORM0 A discount factor 0 ≤ γ < 1 is necessary to exponentially decay the future rewards ensuring bounded returns. While the series is infinite, it is common to use this expression even in the case of possible terminations. Indeed, episode terminations can be considered to be the entering of an absorbing state that transitions only to itself and generates zero rewards thereafter. However, when the maximum length of an episode is predefined, it is easier to rewrite the expression above by explicitly including the time limit T: DISPLAYFORM1 Optimizing for the expectation of the return specified in Equation 2 is suitable for naturally timelimited tasks where the agent has to maximize its expected return G 0:T over a fixed episode length only. In this case, since the return is bounded, a discount factor of γ = 1 can be used. However, in practice it is still common to keep γ smaller than 1 in order to give more priority to short-term rewards. Under this optimality model, the objective of the agent does not go beyond the time limit. Therefore, an agent optimizing under this model should ideally learn to take more risky actions that reveal higher expected return than safer ones as approaching the end of the time limit. In Section 2, we study this case and illustrate that due to the presence of the time limit, the remaining time is present in the environment's state and is essential to its Markov property BID19. Therefore, we propose to include a notion of the remaining time in the agent's input, an approach that we refer to as time-awareness. We describe various general scenarios where lacking a notion of the remaining time can lead to suboptimal policies and instability, and demonstrate significant performance improvements for time-aware agents. Optimizing for the expectation of the return specified by Equation 1 is relevant for time-unlimited tasks where the interaction is not limited in time by nature. In this case, the agent has to maximize its expected return over an indefinite (e.g. infinite) period. However, it is desirable to use time limits in order to diversify the agent's experience. For example, starting from highly diverse states can avoid converging to suboptimal policies that are limited to a fraction of the state space. In Section 3, we show that in order to learn good policies that continue beyond the time limit, it is important to differentiate between the terminations that are due to time limits and those from the environment. Specifically, for value-based algorithms, we propose to continue bootstrapping at states where termination is due to the time limit, or generally any other causes other than the environmental ones. We refer to this method as partial-episode bootstrapping. We describe various scenarios where having a time limit can facilitate learning, but where the aim is to learn optimal policies for indefinite periods, and demonstrate that our method can significantly improve performance. We evaluate the impact of the proposed methods on a range of novel and popular benchmark domains using a deep reinforcement learning BID0 BID9 algorithm called the Proximal Policy Optimization (PPO), one which has recently been used to achieve stateof-the-art performance in many domains BID17 BID8. We use the OpenAI Baselines 1 implementation of the PPO algorithm with the hyperparameters reported by BID17, unless explicitly specified. All novel environments are implemented using the OpenAI Gym framework BID3 and the standard benchmark domains are from the MuJoCo BID22 Gym collection. We modified the TimeLimit wrapper to include remaining time in the observations for the proposed time-aware agent and a flag to separate timeout terminations from environmental ones for the proposed partial-episode bootstrapping agent. For every task involving PPO, to have perfect reproducibility, we used the same 40 seeds from 0 to 39 to initialize the pseudo-random number generators for the agents and environments. Every 5 training cycles (i.e. 10240 time steps), we perform an evaluation on a complete episode and store the sums of rewards, discounted returns, and estimated state-values. For generating the performance plots, we average the values across all runs and then apply smoothing with a sliding window of size 10. The performance graphs show these smoothed averages as well as their standard error. We empirically show that time-awareness significantly improves the performance and stability of PPO for the time-limited tasks and can sometimes in quite interesting behaviors. For example, in the Hopper-v1 domain with T = 300, our agent learns to efficiently jump forward and fall towards the end of its time in order to maximize its travelled distance and achieve a "photo finish". For the time-unlimited tasks, we show that bootstrapping at the end of partial episodes allows to significantly outperform the standard PPO. In particular, on Hopper-v1, even if trained with episodes of only 200 steps, our agent manages to learn to hop for at least 10 6 time steps, ing in more than two hours of rendered video. Detailed for all variants of the tasks using PPO with and without the proposed methods are available in the Appendix. The source code will be made publicly available shortly. A visual depiction of highlights of the learned behaviors can be viewed at the address sites.google.com/view/time-limits-in-rl. In tasks that are time-limited by nature, the learning objective is to optimize the expectation of the return G γ 0:T from Equation 2. Interactions are systematically terminated at a fixed predetermined time step T if no environmental termination occurs earlier. This time-wise termination can be seen as transitioning to a terminal state whenever the time limit is reached. The states of the agent's environment, formally a Markov decision process (MDP) BID14, thus must contain a notion of the remaining time that is used by its transition function. This time-dependent MDP can be thought of as a stack of T time-independent MDPs, one for each time step, followed by one that only transitions to a terminal state. Thus, a decision-making agent in such an environment, at every time step t ∈ {0, ..., T − 1}, takes a decision that in transitioning to a new state from the next MDP in the stack and receiving a reward. Thus, a time-unaware agent effectively has to act in a partially observable Markov decision process (POMDP) BID12 where states that only differ by their remaining time appear identical. This phenomenon is a form of state-aliasing BID24 that is known to lead to suboptimal policies and instability due to the infeasibility of correct credit assignment. In this case, the terminations due to time limits can only be interpreted as part of the environment's stochasticity where the time-unaware agent perceives a chance of transitioning to a terminal state from any given state. In fact, this perceived stochasticity is dynamically changing with the agent's behavioral policy. For example, an agent could choose to stay in a fixed initial state during the entire course of an episode and perceive the probability of termination from that state to be 1 T, whereas it could choose to always move away from it in which case this probability would be perceived to be zero. In the view of the above, we propose time-awareness for reinforcement learning agents in timelimited domains by including directly the remaining time T − t in the agent's representation of the environment's state or by providing a mean to infer it. The importance of the inclusion of a notion of time in time-limited problems was first demonstrated by BID7, but seems to have been overlooked in the design of the benchmark domains and the evaluation of reinforcement learning agents. A major difference between the approach of BID7 and that of ours, however, is that we consider a more general class of time-dependent MDPs where the reward distribution and the transitions can also be time-dependent, preventing the possibility to consider multiple time instances at once as it is the case for the Q T -learning algorithm BID7.Here, we illustrate the issues faced by time-unaware agents by exemplifying the case for value-based methods. The state-value function for a time-aware agent in an environment with time limit T is: DISPLAYFORM0 By denoting an estimate of the state-value function byv π, the temporal-difference (TD) update rule BID18 after transitioning from a state s to a state s and receiving a reward r as a of an action a is given by either of the following expressions conditioned on the time step t: DISPLAYFORM1 The proposed added notion of the remaining time is indicated in bold and blue. A time-unaware agent would be deprived of this information and thus would updatev π (s) with or without bootstrapping from the estimated value of s depending on whether the time limit is reached. Confused by the conflicting updates for estimating the value of the same state, instead of learning an accurate value function, this time-unaware agent learns an approximate average of these inconsistent updates. Figure 2: The color-coded learned action probabilities overlaid on our Queue of Cars problem (black and white indicate 0 and 1, respectively). For each block, the top row represents the dangerous action and the bottom row the safe one. The 9 non-terminal states are represented horizontally. Left: a time-aware PPO agent at various times: the agent learns to optimally select the dangerous action. Right: 5 different instances of the time-unaware PPO agent. If the time limit T is never varied, inclusion of the time t as a notion of the remaining time would be sufficient. However, for more generality we choose to represent the remaining time T −t. In practice, we used the remaining time normalized from 1 to 0, concatenated to the observations provided by the Gym environments by modifying the TimeLimit wrapper. To give a simple example of the learning of an optimal time-dependent policy, we consider an MDP containing two states A and B. The agent always starts in A and has the possibility to choose an action to "stay" in place with no rewards or a "jump" action that transitions it to state B with a reward of +1. However, state B is considered a trap where the only possible action leads to a penalty of −1. The episodes terminate after a fixed number of steps T. The goal of the game is thus to jump at the last moment. For a time-unaware agent, the task is impossible to master for T > 1 and the best feasible policy would be to stay in place, ing in an overall return of 0. In contrast, a time-aware agent can learn to stay in place for T − 1 steps and then jump, scoring an undiscounted sum of rewards of +1. To further illustrate the impact of state-aliasing for time-unaware agents, we consider a deterministic gridworld environment (see FIG0) consisting of two possible goals rewarding 50 for reaching the top-right and 20 for the bottom-left. The agent has 5 actions: to move in cardinal directions or to stay in place. Any movement incurs a penalty of −1 while staying in place generates a reward of 0. Episodes terminate via a timer at T = 3 or if the agent has reached a goal. The initial state is randomly selected for every episode, excluding goal states. For training, we used a tabular Q-learning BID23 with completely random actions, trained until convergence with a decaying learning rate and a discount factor of γ = 0.99.The time-aware agent has a state-action value table for each time step and easily learns the optimal policy which is to go for the closest goal when there is enough time, and to stay in place otherwise. For the time-unaware agent, the greedy values of the cells adjacent to the goal with the terminal reward of 50 converge to 49 and those adjacent to the goal with 20 converge to 19 because of the −1 penalty on every move. Then, since the time limit is T = 3, from each remaining cell, the agent may have between 1 and 3 steps. For 2/3 of the times, the time-unaware agent receives −1 and bootstraps from the successor cell and for 1/3 of the times it receives −1 and experiences termination. Thus, for v(s) = arg max a q(s, a) and N (s) denoting the neighbors of s, for states non adjacent to the goals we have: v(s) = 2/3(−1 + γ max s ∈N (s) v(s)) + 1/3(−1). This learned value function leads to a policy that always tries to go for the closest goal even if there is not enough time. While the final optimal policy does not require time information, this example clearly shows that the confusion during training due to state-aliasing can create a leakage of the values to states that are out of reach. It is worth noting that, Monte Carlo methods such as REINFORCE BID26 BID20 are not susceptible to this leakage as they use complete returns instead of bootstrapping. However, without awareness of the remaining time, Monte Carlo methods would still not be able to learn an optimal policy in many cases, such as the Last Moment problem or the Queue of Cars problem in the subsequent section. An interesting property of time-aware agents is the ability to dynamically adapt to the remaining time that can, for example, be correlated with the current progress of the agent. To illustrate this, we introduce an environment which we call Queue of Cars where the agent controls a vehicle that is held up behind an intermittently moving queue of cars. The agent's goal is to reach an exit located 9 slots away from its starting position. At any time, the agent can choose the "safe" action to stay in the queue which may in advancing to the next slot with 50% probability. Alternatively, it has the possibility to attempt to overtake by a "dangerous" action that even though it has 80% probability to advance, it poses a 10% chance of collision with the oncoming traffic and terminating the episode. The agent receives no rewards unless it reaches its destination to receive a +1 reward. The episode terminates by reaching the destination, running out of time, or colliding during an overtake. In this task, an agent can have a lucky sequence of safe transitions and reach the destination within the time limit without ever needing to attempt an overtake. However, the opposite can also happen in which case the agent would need to overtake the cars to reach its destination in time. Time-unaware agents cannot possibly gauge the necessity to rush and thus can only learn a statistically efficient combination of dangerous and safe actions based on position only. Figure 2 shows this situation for a time-unaware PPO over 5 different runs against a time-aware one that adapts to the remaining time based on its distance to the goal to take more dangerous actions in the face of time insufficiency. A discount factor of γ = 1 was used for both agents. In this section, we evaluate the performance of PPO with and without the remaining time as part of the agent's input on a set of deterministic, continuous control tasks from the OpenAI's MuJoCo Gym benchmarks BID3 BID22 BID4. By default, these environments use predefined time limits and are each reset to a random initial state after an episode termination. FIG1 shows the performance of a time-unaware PPO against a time-aware one, demonstrating that time-awareness significantly improves the performance of PPO. The learned state-values shown for the InvertedPendulum-v1 task (see FIG1) illustrate perfectly the difference between a time- aware agent and a time-unaware one in terms of their estimated expected return as the episode progresses. While time-awareness enables the agent to learn an accurate exponential or linear decay of the expected return with time, the time-unaware agent only learns a constant estimate due to state-aliasing. Figure 4 (left) shows the performance comparisons of PPO with and without time-awareness in the Hopper-v1 domain with time limit T = 300. With a discount rate of 0.99, the standard PPO is initially on par with the time-aware PPO and later starts to plateau. As the agents become better, they start to experience terminations due to the time limit more frequently, at which point the timeunaware agent begins to perceive inconsistent returns for seemingly similar states. The advantage of the time-aware PPO becomes even clearer in the case of a discount rate of 1 where the time-unaware PPO diverges quite drastically. A possible reason is that the time-unaware PPO agent experiences much more significant conflicts as the returns are now the sum of the undiscounted rewards. This is while, the time-aware PPO still continues to perform well as it is able to assign credits appropriately based on the knowledge of the remaining time. Time-awareness does not only help agents by avoiding the conflicting updates. In fact, in naturally time-limited tasks where the agents have to maximize their performance for a limited time, timeaware agents can demonstrate quite interesting ways of ensuring to achieve this objective. FIG2 show the average final pose of the time-aware (middle) and time-unaware (right) agents. We can see that the time-aware agent learns to jump towards the end of its time in order to maximize its expected return, ing in a "photo finish", something that a time-unaware agent cannot accurately achieve. Finally, FIG2 (bottom-right) shows an interesting behavior robustly demonstrated by the timeunaware PPO in the case of γ = 1 that is to actively stay in place, accumulating at least the rewards coming from the bonus for staying alive. In this section, we explored the scenario where the aim is to learn a policy that maximizes the expected return over a limited time. We proposed to include a notion of the remaining time as part of the agent's observation to avoid state-aliasing which can cause suboptimal policies and instability. However, this scenario is not always ideal as there are cases where, even though the agent experiences time limits in its interaction with the environment, the objective is to learn a policy for a time-unlimited task. For instance, as we saw in the Hopper environment, the learned policy that maximizes the return over the T = 300 time steps generally in a photo finish which would lead to a fall and subsequent termination if the simulation was to be extended. Such a policy is not viable if the goal is to learn to move forward for an indefinite period of time. One solution is to not have time limits during training. However, it is often more efficient to instead have short snippets of interactions to expose the agent to diverse experiences. In the next section, we explore this case and propose a method that enables to effectively learn in such domains from partial episodes. In tasks that are not time-limited by nature, the learning objective is to optimize the expectation of the return G γ 0 from Equation 1. While the agent has to maximize its expected return over an indefinite (possibly infinite) period, it is desirable to still use time limits in order to frequently reset the environment and increase the diversity of the agent's experiences. A common mistake, however, is to then consider the terminations due to such time limits as environmental ones. This is equivalent to optimizing for returns G In the case of value-based algorithms, we propose to continue bootstrapping at states where termination is due to the time limit. The state-value function of a policy (from Equation 3) can be rewritten in terms of the time-limited return G γ t:T and the bootstrapped value from the last state v π (S T): DISPLAYFORM0 By denoting an estimate of the state-value function byv π, the temporal-difference update rule after transitioning from a state s to a state s and receiving a reward r as a of an action a is given by either of the following expressions conditioned on the time step t: DISPLAYFORM1 The proposed partial-episode bootstrap is indicated in bold and green. An agent without this modification would updatev π (s) with or without bootstrapping from the estimated value of s depending on whether there is some remaining time or not. Similarly to Equation 4, the conflicting updates for estimating the value of the same state leads to an approximate average of these updates. While this section is related to the previous one, it is somewhat orthogonal. In the previous section, one of the issues was bootstrapping values from states that were out-of-reach, letting the agent falsely believe that more rewards were available after. On the opposite, the problem presented here is when systematic bootstrapping is not performed from states at the time limit and thus, forgetting that more rewards would actually be available thereafter. We revisit the gridworld environment from Section 2.2. While previously the agent's task was to learn an optimal policy for a given time limit, we now consider how an agent can learn a good policy for an indefinite period from partial-episode experiences. We use the same setup as in Section 2.2. Again, we use a tabular Q-learning, but instead of considering terminations due to time limits as environmental ones, we continue bootstrapping from the non-terminal states that are reached at the time limits. This modification allows our agent to learn the time-unlimited optimal policy of always going for the most rewarding goal (see FIG0 . On the other hand, while the standard agent that is not performing the final bootstrapping (see FIG0) had values from out-of-reach cells leaking into its learned value function, these bootstraps did not occur in sufficient proportion with respect to the terminations due to time limits in order to let the agent learn the time-unlimited optimal policy. For the next experiments, we again use PPO but with two key modifications. We modify the Gym's TimeLimit wrapper to not include the remaining time (as needed for Section 2), but instead to include a flag to differentiate the terminations due to the time limit from the environmental ones. We also modify the PPO's implementation to enable continuing to bootstrap when terminations are due to time limits only. This involves modifying the implementation of the generalized advantage estimation (GAE). While GAEs use an exponentially-weighted average of n-step value estimations for bootstrapping that are more complex than the one-step lookahead bootstrapping explained in Equation 6, continuing to bootstrap from the last non-terminal states (i.e. at the end of the partial-episodes) is the only modification required for the proposed approach. 3 during the evaluations. The standard PPO agent degrades drastically after some time. Here, we consider the Hopper-v1 environment from Section 2.4, but instead aim to learn a policy that maximizes the agent's expected return over a time-unlimited horizon. We do not revisit the Reacherv1 and the InvertedPendulum-v1 environments as their extensions to time-unlimited domains is not of particular value-that is, staying at the target position long after the target is reached (Reacher-v1) or maintaining the pendulum's vertical pose long after it is balanced (InvertedPendulum-v1). The aim here is to show that by continuing to bootstrap from episode terminations that are due to time limits only, we are able to learn good policies for time-unlimited domains. FIG4 demonstrates performance evaluations of the standard PPO against one with the proposed partial-episode bootstrapping modification. The agents are trained on time-limited episodes of maximum T = 200 time steps, and are evaluated in the same environment, but with T = 10 6 time steps. We show that the proposed bootstrapping method significantly outperforms the standard PPO. During the evaluations, the standard PPO agent managed to reach a maximum of 7238 time steps on only one of the 40 training seeds, while our agent managed to reach the evaluation time limit of T = 10 6 on 7 occasions. This is quite impressive as this time limit corresponds to more than 2 hours of rendered hopping. The proposed bootstrapping at time-limit terminations was shown to enable our agent to effectively learn a good long-term policy on Hopper-v1. However, it could be argued that since the Hopper-v1 environment always starts in quite similar configurations, the ant policy is overfitted and almost completely cyclic. To demonstrate the ability of our proposed agent in learning non-cyclic policies for time-unlimited domains, we create a novel MuJoCo environment consisting of a torque-controlled ball that has to be used to push a cube to specified target positions. Once the cube has touched the target, the agent is rewarded and the target is moved away from the cube to a new random position. Because the task lacks terminal states, it can continue indefinitely. The objects are surrounded by fixed bounding walls. The inner edge of the walls stops the cube but not the ball in order to let the agent move the cube even if it is in a corner. The movements of the ball are limited to the horizontal plane and to the area defined by the outer edge of the walls. The environment's state representation consists of the objects' coordinates and velocities, and the cube's rotation. The agent receives no rewards unless the cube reaches a target location, at which point the agent receives a reinforcement reward of 1.Due to the absence of reward shaping, reinforcement learning agents are prone to being stuck, unable to learn to solve problems. Thus it is often useful to introduce a time limit during training in order to facilitate learning. We use a training time limit of 50, only sufficient to push the cube to one target location in most cases. The evaluations, however, consisted of 10 3 steps, long enough to allow successfully reaching several targets. FIG4 shows the performance comparison of the standard PPO against one with the proposed modification. An entropy coefficient of 0.03 is used to encourage exploration and higher the chance of reaching a target and experiencing a reinforcement reward. We found this value to yield best performance for both agents among those from the set {0, 0.01, 0.02, 0.03, 0.04, 0.05}. While the performance of the standard PPO degrades significantly after some time, it is clear that bootstrapping at the time limit helps our agent to perform significantly better. The maximum number of targets reached by our agent in a single episode of evaluation (T = 103) was 36 against 21 for the standard PPO. We showed in Section 2 that time-awareness is required for correct credit assignment in domains where the agent has to optimize its performance over a time-limited horizon. However, even without the knowledge of the remaining time, reinforcement learning agents still often manage to perform relatively well. This could be due to several reasons including: If the time limit is sufficiently long that terminations due to time limits are hardly ever experienced-for instance, in the Arcade Learning Environment (ALE) BID1 BID13 BID12. In deep learning BID11 BID15, it is highly common to use a stack of previous observations or recurrent neural networks (RNNs) BID6 to address scenarios with partial observations BID25 ). These solutions may to an extent help when the remaining time is not included as part of the agent's input. However, they are much more complex architectures and are only next-best solutions, while including a notion of the remaining time is quite simple and allows better diagnosis of the learned policies. The proposed approach is quite generic and can potentially be applied to domains with varying time limits where the agent has to learn to generalize as the remaining time approaches zero. In real-world applications such as robotics the proposed approach could easily be adapted by using the real time instead of simulation time steps. In order for the proposed partial-episode bootstrapping in Section 3 to work, as is the case for valuebased methods in general, the agent needs to bootstrap from reliable estimated predictions. This is in general resolved by enabling sufficient exploration. However, when the interactions are limited in time, exploration of the full state-space may not be feasible from some fixed starting states. Thus, a good way to allow appropriate exploration in such domains is to sufficiently randomize the initial states. It is worth noting that the proposed partial-episode bootstrapping is quite generic in that it is not restricted to partial episodes caused only due to time limits. In fact, this approach is valid for any early termination causes. For instance, it is common in the curriculum learning literature to start from near the goal states (easier tasks), and gradually expand to further states (more difficult tasks) BID5. In this case, it can be helpful to stitch the learned values by terminating the episodes and bootstrapping as soon as the agent enters a state that is already well known. Since the proposed methods were shown to enable to better optimize for the time-limited and timeunlimited domains, we believe that they have the potential to improve the performance and stability of a large number of existing reinforcement learning algorithms. We also propose that, since reinforcement learning agents are in fact optimizing for the expected returns, and not the undiscounted sum of rewards, it is more appropriate to consider this measure for performance evaluation. We considered the problem of learning optimal policies in time-limited and time-unlimited domains using time-limited interactions. We showed that when learning policies for time-limited tasks, it is important to include a notion of the remaining time as part of the agent's input. Not doing so can cause state-aliasing which in turn can in suboptimal policies, instability, and slower convergence. We then showed that, when learning policies that are optimal for time-unlimited tasks, it is more appropriate to continue bootstrapping at the end of the partial episodes when termination is due to time limits, or any other early termination causes other than environmental ones. In both cases, we illustrated that our proposed methods can significantly improve the performance of PPO and allow to optimize more directly, and accurately, for either of the optimality models. Reacher-v1, = 1.0, training limit = 50, evaluation limit = 50

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We consider the problem of learning optimal policies in time-limited and time-unlimited domains using time-limited interactions.

Represent the short summary for finding the corresponding scientific article (long).

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.


Question: Represent the scientific article to find a short TLDR

Question: Although stochastic gradient descent (SGD) is a driving force behind the recent success of deep learning, our understanding of its dynamics in a high-dimensional parameter space is limited. In recent years, some researchers have used the stochasticity of minibatch gradients, or the signal-to-noise ratio, to better characterize the learning dynamics of SGD. Inspired from these work, we here analyze SGD from a geometrical perspective by inspecting the stochasticity of the norms and directions of minibatch gradients. We propose a model of the directional concentration for minibatch gradients through von Mises-Fisher (VMF) distribution, and show that the directional uniformity of minibatch gradients increases over the course of SGD. We empirically verify our using deep convolutional networks and observe a higher correlation between the gradient stochasticity and the proposed directional uniformity than that against the gradient norm stochasticity, suggesting that the directional statistics of minibatch gradients is a major factor behind SGD. Stochastic gradient descent (SGD) has been a driving force behind the recent success of deep learning. Despite a series of work on improving SGD by incorporating the second-order information of the objective function BID26 BID21 BID6 BID22 BID7, SGD is still the most widely used optimization algorithm for training a deep neural network. The learning dynamics of SGD, however, has not been well characterized beyond that it converges to an extremal point BID1 due to the non-convexity and highdimensionality of a usual objective function used in deep learning. Gradient stochasticity, or the signal-to-noise ratio (SNR) of the stochastic gradient, has been proposed as a tool for analyzing the learning dynamics of SGD. BID28 identified two phases in SGD based on this. In the first phase, "drift phase", the gradient mean is much higher than its standard deviation, during which optimization progresses rapidly. This drift phase is followed by the "diffusion phase", where SGD behaves similarly to Gaussian noise with very small means. Similar observations were made by BID18 and BID4 who have also divided the learning dynamics of SGD into two phases. BID28 have proposed that such phase transition is related to information compression. Unlike them, we notice that there are two aspects to the gradient stochasticity. One is the L 2 norm of the minibatch gradient (the norm stochasticity), and the other is the directional balance of minibatch gradients (the directional stochasticity). SGD converges or terminates when either the norm of the minibatch gradient vanishes to zeros, or when the angles of the minibatch gradients are uniformly distributed and their non-zero norms are close to each other. That is, the gradient stochasticity, or the SNR of the stochastic gradient, is driven by both of these aspects, and it is necessary for us to investigate not only the holistic SNR but also the SNR of the minibatch gradient norm and that of the minibatch gradient angles. In this paper, we use a von Mises-Fisher (vMF hereafter) distribution, which is often used in directional statistics BID20, and its concentration parameter κ to characterize the directional balance of minibatch gradients and understand the learning dynamics of SGD from the perspective of directional statistics of minibatch gradients. We prove that SGD increases the direc-tional balance of minibatch gradients. We empirically verify this with deep convolutional networks with various techniques, including batch normalization BID12 and residual connections BID9, on MNIST and CIFAR-10 (BID15). Our empirical investigation further reveals that the proposed directional stochasticity is a major drive behind the gradient stochasticity compared to the norm stochasticity, suggesting the importance of understanding the directional statistics of the stochastic gradient. Contribution We analyze directional stochasticity of the minibatch gradients via angles as well as the concentration parameter of the vMF distribution. Especially, we theoretically show that the directional uniformity of the minibatch gradients modeled by the vMF distribution increases as training progresses, and verify this by experiments. In doing so, we introduce gradient norm stochasticity as the ratio of the standard deviation of the minibatch gradients to their expectation and theoretically and empirically show that this gradient norm stochasticity decreases as the batch size increases. Related work Most studies about SGD dynamics have been based on two-phase behavior BID28 BID18 BID4. BID18 investigated this behavior by considering a shallow neural network with residual connections and assuming the standard normal input distribution. They showed that SGD-based learning under these setups has two phases; search and convergence phases. BID28 on the other hand investigated a deep neural network with tanh activation functions, and showed that SGD-based learning has drift and diffusion phases. They have also proposed that such SNR transition (drift + diffusion) is related to the information transition divided into empirical error minimization and representation compression phases. have reported that the information transition is not generally associated with the SNR transition with ReLU BID23 ) activation functions. BID4 instead looked at the inner product between successive minibatch gradients and presented transient and stationary phases. Unlike our work here, the experimental verification of the previous work conducted under limited settings -the shallow network BID18, the specific activation function BID28, and only MNIST dataset BID28 BID4 -that conform well with their theoretical assumptions. Moreover, their work does not offer empirical about the effect of the latest techniques including both batch normalization BID12 layers and residual connections BID9. Norms and Angles Unless explicitly stated, a norm refers to L 2 norm. · and ·, · thus correspond to L 2 norm and the Euclidean inner product on R d, respectively. We use x n ⇒ x to indicate that "a random variable x n converges to x in distribution." Similarly, x n P → x means convergence in probability. An angle θ between d-dimensional vectors u and v is defined by θ = Loss functions A loss function of a neural network is written as f (w) = 1 n n i=1 f i (w), where w ∈ R d is a trainable parameter. f i is "a per-example loss function" computed on the i-th data point. We use I and m to denote a minibatch index set and its batch size, respectively. Further, we call f I (w) = 1 m i∈I f i (w) "a minibatch loss function given I". In Section 3.1, we use g i (w) and g(w) to denote −∇ w f i (w) and −∇ w f I (w), respectively. In Section 3.3, the index i is used for the corresponding minibatch index set I i. For example, the negative gradient of f Ii (w) is written aŝ g i (w). During optimization, we denote a parameter w at the i-th iteration in the t-th epoch as w Figure 1: Characteristics of the vMF distribution in a 2-dimensional space. 100 random samples are drawn from vMF(µ, κ) where µ = and κ = {0, 5, 50}.on the hypersphere DISPLAYFORM0 Here, the concentration parameter κ determines how the samples from this distribution are concentrated on the mean direction µ and C d (κ) is constant determined by d and κ. If κ is zero, then it is a uniform distribution on the unit hypersphere, and as κ → ∞, it becomes a point mass on the unit hypersphere (Figure 1). The maximum likelihood estimates for µ and κ arê DISPLAYFORM1 1−r 2 where x i's are random samples from the vMF distribution andr = n i=1 xi n. The formula forκ is approximate since the exact computation is intractable BID0. It is a usual practice for SGD to use a minibatch gradientĝ(w) = −∇ w f I (w) instead of a full batch gradient g(w) = −∇ w f (w). The minibatch index set I is drawn from {1,. . .,n} randomly.ĝ(w) satisfies E[ĝ(w)] = g(w) and Cov(ĝ(w),ĝ(w)) ≈ 1 mn n i=1 g i (w)g i (w) for n m where n is the number of full data points and g i (w) = −∇ w f i (w) BID11. As the batch size m increases, the randomness ofĝ(w) decreases. Hence E ĝ(w) tends to g(w), and Var(ĝ(w) ), which is the variance of the norm of the minibatch gradient, vanishes. The convergence rate analysis is as the following: Theorem 1. Letĝ(w) be a minibatch gradient induced from the minibatch index set I of batch size m from {1, . . ., n} and suppose γ = max i,j∈{1,...,n} | g i (w), g j (w) |. Then DISPLAYFORM0 and DISPLAYFORM1 Proof. See Supplemental A.According to Theorem 1, a large batch size m reduces the variance of ĝ(w) centered at E ĝ(w) with convergence rate O(1/m). We empirically verify this by estimating the gradient norm stochasticity at random points while varying the minibatch size, using a fully-connected neural network (FNN) with MNIST, as shown in FIG2 (a) (see Supplemental E for more details.) This theorem however only demonstrate that the gradient norm stochasticity is (l.h.s. of FORMULA3) is low at random initial points. It may blow up after SGD updates, since the upper bound (r.h.s. of FORMULA3) is inversely proportional to g(w). This implies that the learning dynamics and convergence of SGD, measured in terms of the vanishing gradient, i.e., n b i=1ĝ i (w) ≈ 0, is not necessarily explained by the vanishing norms of minibatch gradients, but rather by the balance of the directions ofĝ i (w)'s, which motivates our investigation of the directional statistics of minibatch gradients. See FIG2 (b) as an illustration. In order to investigate the directions of minibatch gradients and how they balance, we start from an angle between two vectors. First, we analyze an asymptotic behavior of angles between uniformly random unit vectors in a high-dimensional space. Theorem 2. Suppose that u and v are mutually independent d-dimensional uniformly random unit vectors. Then, DISPLAYFORM0 Proof. See Supplemental B.According to Theorem 2, the angle between two independent uniformly random unit vectors is normally distributed and becomes increasingly more concentrated as d grows FIG3 ). If SGD iterations indeed drive the directions of minibatch gradients to be uniform, then, at least, the distribution of angles between minibatch gradients and a given uniformly sampled unit vector follows asymptotically DISPLAYFORM1 Figures 3(b) and 3(c) show that the distribution of the angles between minibatch gradients and a given uniformly sampled unit vector converges to an asymptotic distribution after SGD iterations. Although we could measure the uniformity of minibatch gradients how the angle distribution between minibatch gradients is close to, it is not as trivial to compare the distributions as to compare numerical values. This necessitates another way to measure the uniformity of minibatch gradients. We draw a density plot θ(u,ĝ j (w)) ĝ j (w)) ) for 3, 000 minibatch gradients (black) at w = w 0 0 (b) and w = w 0 final, with training accuracy of > 99.9%, (c) when u is given. After SGD iterations, the density of θ(u,ĝ j (w)) converges to an asymptotic density (red). The dimension of FNN is 635,200. To model the uniformity of minibatch gradients, we propose to use the vMF distribution in Definition 1. The concentration parameter κ measures how uniformly the directions of unit vectors are distributed. By Theorem 1, with a large batch size, the norm of minibatch gradient is nearly deterministic, andμ is almost parallel to the direction of full batch gradient. In other words, κ measures the concentration of the minibatch gradients directions around the full batch gradient. The following Lemma 1 introduces the relationship between the norm of averaged unit vectors and κ, the approximate estimator of κ. Lemma 1. The approximated estimator of κ induced from the d-dimensional unit vectors DISPLAYFORM0 Moreover, h(·) and h (·) are strictly increasing and increasing on [0, n b), respectively. Proof. See Supplemental C.1. DISPLAYFORM1 which is measured from the directions from the current location w to the fixed points p i's, where h(·) is a function defined in Lemma 1. Since h(·) is an increasing function, we may focus only on DISPLAYFORM2 pi−w pi−w to see howκ behaves with respect to its argument. Lemma 2 implies that the estimated directional concentrationκ decreases if we move away from w FIG5 ). In other words,κ(w) <κ(w 0 0). DISPLAYFORM3 If all p i's are not on a single ray from the current location w, then there exists positive number η such that DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 Proof. See Supplemental C.2.We make the connection between the observation above and SGD by first viewing p i's as local minibatch solutions. Definition 2. For a minibatch index set I i, p i (w) = arg min w ∈N (w;ri) f Ii (w) is a local minibatch solution of I i at w, where N (w; r i) is a neighborhood of radius r i at w. Here, r i is determined by w and I i for p i (w) to exist uniquely. Under this definition, p i (w) is local minimum of a minibatch loss function f Ii near w. Then we reasonably expect that the direction ofĝ i (w) = −∇ w f Ii (w) is similar to that of p i (w) − w. Each epoch of SGD with a learning rate η computes a series of w DISPLAYFORM7.., n b } with a large batch size or at the early stage of SGD iterations. Combining these approximations, ĝ i (w DISPLAYFORM8 For example, suppose that t = 0, n b = 3 and τ = 1, and assume that p i (w DISPLAYFORM9 Hence, we haveκ(w DISPLAYFORM10 DISPLAYFORM11 for a sufficiently small ξ > 0, then there exists positive number η such that DISPLAYFORM12 Proof. See Supplemental C.3.This Theorem 3 asserts thatκ(·) decreases even with some perturbation along the averaged direction Without the corollary above, we need to solve p i (w 0 t) = arg min w∈N (w 0 t ;r) f Ii (w) for all i ∈ {1, . . ., n s}, where n s is the number of samples to estimate κ, in order to computeκ(w 0 t). Corollary 3.1 however implies that we can computeκ(w 0 t) by usingĝ DISPLAYFORM13 In Practice Although the number of all possible minibatches in each epoch is n b = n m, it is often the case to use n b ≈ n/m minibatches at each epoch in practice to go from w 0 t to w 0 t+1. Assuming that these n b minibatches were selected uniformly at random, the average of the n b normalized minibatch gradients is the maximum likelihood estimate of µ, just like the average of all n b normalized minibatch gradients. Thus, we expect with a large n b, DISPLAYFORM14 and that SGD in practice also satisfiesκ(w In order to empirically verify our theory on directional statistics of minibatch gradients, we train various types of deep neural networks using SGD and monitor the following metrics for analyzing the learning dynamics of SGD:• Training loss Figure 5: We show the averageκ (black curve) ± std. (shaded area), as the function of the number of training epochs (in log-log scale) across various batch sizes in MNIST classifications using FNN with fixed learning rate 0.01 and 5 random initializations. Althoughκ with the large batch size decreases more smoothly rather than the small batch size, we observe thatκ still decreases well with minibatches of size 64. We did not match the ranges of the y-axes across the plots to emphasize the trend of monotonic decrease.• Validation loss DISPLAYFORM0 The latter three quantities are statistically estimated using n s = 3, 000 minibatches. We useκ to denote the κ estimate. We train the following types of deep neural networks (Supplemental E):• FNN: a fully connected network with a single hidden layer • DFNN: a fully connected network with three hidden layers • CNN: a convolutional network with 14 layers BID16 In the case of the CNN, we also evaluate its variant with skip connections (+Res) BID9.As it was shown recently by BID27 that batch normalization BID12 improves the smoothness of a loss function in terms of its Hessian, we also test adding batch normalization to each layer right before the ReLU BID23 ) nonlinearity (+BN). We use MNIST for the FNN, DFNN and their variants, while CIFAR-10 (BID15 for the CNN and its variants. Our theory suggests a sufficiently large batch size for verification. We empirically analyze how large a batch size is needed in Figure 5 . From these plots,κ decreases monotonically regardless of the minibatch size, but the variance over multiple training runs is much smaller with a larger minibatch size. We thus decide to use a practical size of 64. With this fixed minibatch size, we use a fixed learning rate of 0.01, which allows us to achieve the training accuracy of > 99.9% for every training run in our experiments. We repeat each setup five times starting from different random initial parameters and report both the mean and standard deviation. FNN and DFNN We first observe thatκ decreases over training regardless of the network's depth in FIG12 (a,b). We however also notice thatκ decrease monotonically with the FNN, but less so with its deeper variant (DFNN). We conjecture this is due to the less-smooth loss landscape of a deep neural network. This difference between FNN and DFNN however almost entirely vanishes when batch normalization (+BN) is applied (FIG12 (e,f)). This was expected as batch normalization is known to make the loss function behave better, and our theory assumes a smooth objective function. CNN The CNN is substantially deeper than either FNN or DFNN and is trained on a substantially more difficult problem of CIFAR-10. In other words, the assumptions underlying our theory may not hold as well. Nevertheless, as shown in FIG12 Effect of +BN and +Res Based on our observations that the uniformity of minibatch gradients increases monotonically, when a deep neural network is equipped with residual connection (+Res) and trained with batch normalization (+BN), we conjecture that the loss function induced from these two techniques better satisfies the assumptions underlying our theoretical analysis, such as its well-behavedness. This conjecture is supported by for instance BID27, who demonstrated batch normalization guarantees the boundedness of Hessian, and BID25, who showed residual connections eliminate some singularities of Hessian.κ near the end of training The minimum averageκ of DFNN+BN, which has 1, 920, 000 parameters, is 71, 009.20, that of FNN+BN, which has 636, 800 parameters, is 23, 059.16, and that of CNN+BN+Res, which has 207, 152 parameters, is 20, 320.43. These averageκ are within a constant multiple of estimated κ using 3,000 samples from the vMF distribution with true κ = 0 (35, 075.99 with 1, 920, 000 dimensions, 11, 621.63 with 636, 800 dimensions, and 3, 781.04 with 207, 152 dimensions.) This implies that we cannot say that the underlying directional distribution of minibatch gradients in all these cases at the end of training is not close to uniform BID5. For more detailed analysis, see Supplementary F. The gradient stochasticity (GS) was used by BID28 as a main metric for identifying two phases of SGD learning in deep neural networks. This quantity includes both the gradient norm stochasticity (GNS) and the directional uniformity κ, implying that either or both of GNS and κ could drive the gradient stochasticity. We thus investigate the relationship among these three quantities as well as training and validation losses. We focus on CNN, CNN+BN and CNN+Res+BN trained on CIFAR-10. and directional uniformityκ. We normalized each quantity by its maximum value over training for easier comparison on a single plot. In all the cases, SNR (orange) andκ (red) are almost entirely correlated with each other, while normSNR is less correlated. (Second row) We further verify this by illustrating SNR-κ scatter plots (red) and SNR-normSNR scatter plots (blue) in log-log scales. These plots suggest that the SNR is largely driven by the directional uniformity. From FIG13 (First row), it is clear that the proposed metric of directional uniformityκ correlates better with the gradient stochasticity than the gradient norm stochasticity does. This was especially prominent during the early stage of learning, suggesting that the directional statistics of minibatch gradients is a major explanatory factor behind the learning dynamics of SGD. This difference in correlations is much more apparent from the scatter plots in FIG13 (Second row). We show these plots created from other four training runs per setup in Supplemental G. Stochasticity of gradients is a key to understanding the learning dynamics of SGD BID28 and has been pointed out as a factor behind the success of SGD (see, e.g., BID17 BID14 . In this paper, we provide a theoretical framework using von Mises-Fisher distribution, under which the directional stochasticity of minibatch gradients can be estimated and analyzed, and show that the directional uniformity increases over the course of SGD. Through the extensive empirical evaluation, we have observed that the directional uniformity indeed improves over the course of training a deep neural network, and that its trend is monotonic when batch normalization and skip connections were used. Furthermore, we demonstrated that the stochasticity of minibatch gradients is largely determined by the directional stochasticity rather than the gradient norm stochasticity. Our work in this paper suggests two major research directions for the future. First, our analysis has focused on the aspect of optimization, and it is an open question how the directional uniformity relates to the generalization error although handling the stochasticity of gradients has improved SGD BID24 BID11 BID29 BID13 . Second, we have focused on passive analysis of SGD using the directional statistics of minibatch gradients, but it is not unreasonable to suspect that SGD could be improved by explicitly taking into account the directional statistics of minibatch gradients during optimization. In proving Theorem 1, we use Lemma A.1. Define selector random variables BID11 as below: DISPLAYFORM0 Then we haveĝ DISPLAYFORM1 Lemma A.1. Letĝ(w) be a minibatch gradient induced from the minibatch index set I with batch size m from {1, . . ., n}. Then DISPLAYFORM2 where γ = max i,j∈{1,...,n} | g i (w), g j (w) |. DISPLAYFORM3 Theorem 1. Letĝ(w) be a minibatch gradient induced from the minibatch index set I of batch size m from {1, . . ., n} and suppose γ = max i,j∈{1,...,n} | g i (w), g j (w) |. Then DISPLAYFORM4 and DISPLAYFORM5 Hence, Var(ĝ(w) ) DISPLAYFORM6 2 ≤ E ĝ(w) 2. From the second inequality and Lemma A.1, DISPLAYFORM0. DISPLAYFORM1 For proofs, Slutsky's theorem and delta method are key to describe limiting behaviors of random variables in distributional sense. Theorem B.1. (Slutsky's theorem, BID3) Let {x n}, {y n} be a sequence of random variables that satisfies x n ⇒ x and y n P → ρ when n goes to infinity and ρ is constant. Then x n y n ⇒ cx Theorem B.2. (Delta method, BID3) Let y n be a sequence of random variables that satisfies √ n(y n − µ) ⇒ N (0, σ 2). For a given smooth function f: R → R, suppose that f (µ) exists and is not 0 where f is a derivative. Then DISPLAYFORM0 Lemma B.1. Suppose that u and v are mutually independent d-dimensional uniformly random unit vectors. Then, DISPLAYFORM1 Proof. Note that d-dimensional uniformly random unit vectors u can be generated by normalization of d-dimensional multivariate standard normal random vectors x ∼ N (0, DISPLAYFORM2 Suppose that two independent uniformly random unit vector u and v are generated by two indepen- DISPLAYFORM3 By SLLN, we have DISPLAYFORM4 Since almost sure convergence implies convergence in probability, DISPLAYFORM5 Therefore, by Theorem B.1 (Slutsky's theorem), DISPLAYFORM6 Theorem 2. Suppose that u and v are mutually independent d-dimensional uniformly random unit vectors. Then, DISPLAYFORM7 Proof. Suppose that µ = 0, σ = 1, and f (·) = DISPLAYFORM8 Moreover, h(·) and h (·) are strict increasing and increasing on [0, n b), respectively. (1 −r 2) 2 and its numerator is always positive for d > 2. When d = 2, DISPLAYFORM0 (1 −r 2) 2 > 0. Soκ increases asr increases. The Lipschitz continuity of h(·) directly comes from the continuity of DISPLAYFORM1 Recall that any continuous function on the compact interval [0, n b (1 −)] is bounded. Hence the derivative ofκ with respect to u is bounded. This implies the Lipschitz continuity of h(·).h(·) is strictly increasing sincer = u n b. Further, DISPLAYFORM2 If all p i's are not on a single ray from the current location w, then there exists positive number η such that DISPLAYFORM3 Proof. Without loss of generality, we regard w as the origin. DISPLAYFORM4. Therefore, we only need to show DISPLAYFORM5 we have DISPLAYFORM6 Note that p j = p j and x j = 1. We have DISPLAYFORM7 Since the equality holds when u, x j 2 = u 2 x j 2 for all j, we have strict inequality when all p i's are not located on a single ray from the origin. The proof of Theorem 3 is very similar to that of Lemma 2. DISPLAYFORM0 for sufficiently small ξ > 0, then there exists positive number η such that DISPLAYFORM1 for all ∈ (0, η].Proof. We regard w 0 t as the origin 0. For simplicity, write DISPLAYFORM2 Now we differentiatef with respect to, that is, DISPLAYFORM3 Recall thatp j = p j. Rewrite pj pj = x j and use f in the proof of Lemma 2 DISPLAYFORM4 Since f < 0 by the proof of Lemma 2, DISPLAYFORM5 By using x j = 1 and applying the Cauchy inequality, DISPLAYFORM6 Define r = min j p j. If ) where η is a learning rate. To prove Corollary 3.1, we need to showκ(w 0 t+1) <κ(w 0 t) which is equivalent to DISPLAYFORM7 DISPLAYFORM8 Since DISPLAYFORM9 is Lipschitz continuous on R BID2. If the batch size is sufficiently large and the learning rate η is sufficiently small, ĝ i (w DISPLAYFORM10 If we denote τ η as, we can convert to. DISPLAYFORM11 Since both w DISPLAYFORM12 where σ max (A) and σ min (A) are maximal and minimal singular values of A, respectively. If A is positive-definite matrix, then DISPLAYFORM13 Here λ max (A) and λ min (A) are maximal and minimal eigenvalues of A, respectively. Lemma D.1. If the condition number of the positive definite Hessian matrix of f Ii at a local minibatch solution p i, denoted by H i = ∇ w 2 f Ii (p i), is close to 1 (well-conditioned), then the direction to p i from w is approximately parallel to its negative gradient at w. That is, for all w ∈ R, DISPLAYFORM14 Proof. By the second order Taylor expansion, DISPLAYFORM15 Then, we only need to show DISPLAYFORM16 Since H i is positive definite, we can diagonalize it as H i = P i Λ i P i where P i is an orthonormal transition matrix for H i. DISPLAYFORM17 for sufficiently small ξ. This implies since DISPLAYFORM18.Then we can apply Theorem 3 andκ(w DISPLAYFORM19 where h(·) is increasing and Lipschitz continuous(Lemma 1). By Lemma D.1, we have DISPLAYFORM20 t ) where rhs is bounded by ξ. Hence, Lipschitz continuity of h(·) implies that DISPLAYFORM21 Since t is arbitrary, we can apply this for all w ∈ R including w 0 t+1. For all cases, their weighted layers do not have biases, and dropout BID30 is not applied. We use Xavier initializations BID8 and cross entropy loss functions for all experiments. FNN The FNN is a fully connected network with a single hidden layer. It has 800 hidden units with ReLU BID23 activations and a softmax output layer. DFNN The DFNN is a fully connected network with three hidden layers. It has 800 hidden units with ReLU activations in each hidden layers and a softmax output layer. CNN The network architecture of CNN is similar to the network introduced in BID10 as a CIFAR-10 plain network. The first layer is 3 × 3 convolution layer and the number of output filters are 16. After that, we stack of {4, 4, 3, 1} layers with 3 × 3 convolutions on the feature maps of sizes {32, 16, 8, 4} and the numbers of filters {16, 32, 64, 128}, respectively. The subsampling is performed with a stride of 2. All convolution layers are activated by ReLU and the convolution part ends with a global average pooling BID19, a 10-way fully-conneted layers, and softmax. Note that there are 14 stacked weighted layers.+BN We apply batch normalization right before the ReLU activations on all hidden layers.+Res The identity skip connections are added after every two convolution layers before ReLU nonlinearity (After batch normalization, if it is applied on it.). We concatenate zero padding slices backwards when the number of filters increases. We use neither data augmentations nor preprocessings except scaling pixel values into both MNIST and CIFAR-10. In the case of CIFAR-10, for validation, we randomly choose 5000 images out of 50000 training images. Figure 8: We showκ estimated from {1, 000 (black), 2, 000 (blue), 3, 000 (red)} random samples of the vMF distribution with underlying true κ in 10, 000-dimensional space, as the function of κ (in log-log scale except 0). For large κ, it is well-estimated byκ regardless of sample sizes. When the true κ approaches 0, we need a larger sample size to more accurately estimate this. 635, 200 20, 111.90 ± 13.04 14, 196.89 ± 14.91 11, 607.39 ± 9.27 FNN+BN 636, 800 20, 157.57 ± 14.06 14, 259.83 ± 16.38 11, 621.63 ± 6.83 DFNN 1, 915, 200 60, 619.02 ± 13.49 42, 849.86 ± 18.90 34, 983.31 ± 15.62 DFNN+BN 1, 920, 000 60, 789.84 ± 17.93 42, 958.71 ± 25.61 35, 075.99 ± 12.39 F SOME NOTES ABOUT THE κ ESTIMATE We point out that, for a small κ, the absolute value ofκ is not a precise indicator of the uniformity due to its dependence on the dimensionality, as was investigated earlier by BID5. In order to verify this claim, we run some simulations. First, we vary the number of samples and the true underlying κ with the fixed dimensionality (Unfortunately, we could not easily go over 10, 000 dimensions due to the difficulty in sampling from the vMF distribution with positive κ.). We draw {1, 000, 2, 000, 3, 000} random samples from the vMF distribution with the designated κ. We computeκ from these samples. As can be seen from Figure 8, theκ approaches the true κ from above as the number of samples increases. When the true κ is large, the estimation error rapidly becomes zero as the number of samples approaches 3, 000. When the true κ is low, however, the gap does not narrow completely even with 3, 000 samples. While fixing the true κ to 0 and the number of samples to {1, 000, 2, 000, 3, 000}, we vary the dimensionality to empirically investigate theκ. We choose to use 3, 000 samples to be consistent with our experiments in this paper. We run five simulations each and report both mean and standard deviation TAB1.We clearly observe the trend of increasingκ's with respect to the dimensions. This suggests that we should not compare the absolute values ofκ's across different network architectures due to the differences in the number of parameters. This agrees well with BID5 which empirically showed that the threshold for rejecting the null hypothesis of κ = p by usingκ where p is a fixed value grows with respect to the dimensions. We show plots from other four training runs in FIG12. For all runs, the curves of GS (inverse of SNR) andκ are strongly correlated while GNS (inverse of normSNR) is less correlated to GS. Figure 9: (a,c,e) We plot the evolution of the training loss (Train loss), validation loss (Valid loss), inverse of gradient stochasticity (SNR), inverse of gradient norm stochasticity (normSNR) and directional uniformity κ. We normalized each quantity by its maximum value over training for easier comparison on a single plot. In all the cases, SNR (orange) andκ (red) are almost entirely correlated with each other, while normSNR is less correlated. (b,d,f) We further verify this by illustrating SNR-κ scatter plots (red) and SNR-normSNR scatter plots (blue) in log-log scales. These plots suggest that the SNR is largely driven by the directional uniformity. Figure 10: (a,c,e) We plot the evolution of the training loss (Train loss), validation loss (Valid loss), inverse of gradient stochasticity (SNR), inverse of gradient norm stochasticity (normSNR) and directional uniformity κ. We normalized each quantity by its maximum value over training for easier comparison on a single plot. In all the cases, SNR (orange) andκ (red) are almost entirely correlated with each other, while normSNR is less correlated. (b,d,f) We further verify this by illustrating SNR-κ scatter plots (red) and SNR-normSNR scatter plots (blue) in log-log scales. These plots suggest that the SNR is largely driven by the directional uniformity. Under review as a conference paper at ICLR 2019 Figure 11: (a,c,e) We plot the evolution of the training loss (Train loss), validation loss (Valid loss), inverse of gradient stochasticity (SNR), inverse of gradient norm stochasticity (normSNR) and directional uniformityκ. We normalized each quantity by its maximum value over training for easier comparison on a single plot. In all the cases, SNR (orange) and κ (red) are almost entirely correlated with each other, while normSNR is less correlated. (b,d,f) We further verify this by illustrating SNR-κ scatter plots (red) and SNR-normSNR scatter plots (blue) in log-log scales. These plots suggest that the SNR is largely driven by the directional uniformity. Figure 12: (a,c,e) We plot the evolution of the training loss (Train loss), validation loss (Valid loss), inverse of gradient stochasticity (SNR), inverse of gradient norm stochasticity (normSNR) and directional uniformity κ. We normalized each quantity by its maximum value over training for easier comparison on a single plot. In all the cases, SNR (orange) andκ (red) are almost entirely correlated with each other, while normSNR is less correlated. (b,d,f) We further verify this by illustrating SNR-κ scatter plots (red) and SNR-normSNR scatter plots (blue) in log-log scales. These plots suggest that the SNR is largely driven by the directional uniformity.

represent the tldr for finding the corresponding scientific article

One of theoretical issues in deep learning

Represent the short summary for finding the corresponding scientific article (long)

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO