Question: Represent this text

Question: Abstract Stochastic gradient descent (SGD) and Adam are commonly used to optimize deep neural networks, but choosing one usually means making tradeoffs between speed, accuracy and stability. Here we present an intuition for why the tradeoffs exist as well as a method for unifying the two in a continuous way. This makes it possible to control the way models are trained in much greater detail. We show that for default parameters, the new algorithm equals or outperforms SGD and Adam across a range of models for image classification tasks and outperforms SGD for language modeling tasks. One of the most common methods of training neural networks is stochastic gradient descent (SGD) . SGD has strong theoretical guarantees, including convergence in locally non-convex optimization problems . It also shows improved generalization and stability when compared to other optimization algorithms . There have been various efforts in improving the speed and generalization of SGD. One popular modification is to use an adaptive gradient , which scales the gradient step size to be larger in directions with consistently small gradients. Adam, an implementation that combines SGD with momentum and an adaptive step size inversely proportional to the RMS gradient, has been particularly successful at speeding up training and solving particular problems . However, at other problems it pays a penalty in worse generalization , and it requires additional modifications to achieve a convergence guarantee . Here we develop an intuition for adaptive gradient methods that allows us to unify Adam with SGD in a natural way. The new optimizer, SoftAdam, descends in a direction that mixes the SGD with Adam update steps. As such, it should be able to achieve equal or better optimization across a variety of problems. Several authors have recently tried to combine Adam and SGD to get the best of both worlds. However, these have not always enabled better generalization or performance across different problems. In one study, the optimization algorithm was switched from Adam to SGD during training based on a scale-free criterion, preventing the addition of a new hyper-parameter . The is that the longer the convolutional networks were trained on Adam, the worse their generalization performance compared to SGD. The best performance came from switching to SGD in this case as soon as possible. Another recent algorithm takes the approach of clipping the large Adam updates to make them more similar to SGD as the training nears the end . However, this approach requires two new hyper-parameters: the rate at which the training is switched over, and a learning rate for both SGD and Adam. Similar to this work, partially adaptive methods can allow arbitrary mixing between SGD and Adam. However, in that work the step size is not strictly smaller than the SGD step and so the same guarantees cannot be made about convergence. It is of interest to see whether there is any advantage over these methods. Other algorithms have shown improvements on SGD by averaging weights over many steps (; ;). These algorithms are complementary to the algorithm developed here, as they require an underlying algorithm to choose the step direction at any point. The fundamental idea of gradient descent is to follow the path of steepest descent to an optimum. Stochastic gradient descent enables us to optimize much larger problems by using randomly subsampled training data. The stochastic gradient descent algorithm will minimize the loss function J(θ; x), which is parameterized by θ and takes as input training data x, where α is a learning rate that may vary with t and x t is the training data selected for the batch at step t. The convergence rate can be improved further by using a running average of the gradient, initializing with m 0 ← 0. This method, known as momentum , may be written as, A further development that has improved convergence, especially for LSTM and language modeling tasks, involves the second gradient as well. This specific version is known as the Adam algorithm , are unbiased estimators of the first and second moment respectively. In order to analyze the convergence of these algorithms, we can consider a second-order approximation of J on its combined argument z = (θ; x) in the region of (θ t ; x t), where H t is the Hessian of J(z) around z t. This gives us the gradient, which becomes the SGD update step, Unrolling this update step can be shown to lead to an expression for the distance from the optimal value z, in the basis of Hessian eigenvectors ξ i: We can see that the learning is stable if the learning rate α satisfies, In addition, we find that the value for the learning rate that leads to the fastest overall convergence is, where λ 1 and λ n are the max and min eigenvalues of H, respectively. If rather than a single learning rate α, we were to use a diagonal matrix D such that the update is, we may be able to modify the diagonal entries such that faster overall convergence is achieved. For example, in the special case that the Hessian is diagonal, the convergence rate for the i-th element becomes, i. In this situation, if the eigenvalues λ i are known, the algorithm can converge to the minimum in exactly one step. This corresponds with some intuition behind adaptive moment methods: that taking a step with a "constant" size in every direction toward the target will reach convergence faster than taking a step proportional to the gradient size. Because the eigenvalues and eigenvectors not known a priori, for a practical algorithm we must rely on an approximation to find d i. One technique named AdaGrad prescribes the diagonal elements: For building our intuition, we consider the special case where the Hessian is diagonal, Combining this with Equation 2, we compare the AdaGrad coefficient to the optimal value for As long as αλ i, this will be true when, This may be true on average if the initializations z 0i and optima z i can be made over the same subspaces. That is, if z i is uncorrelated to λ i, we can expect this to have good performance on average. However, there can be significant errors in both overestimating and underestimating the eigenvalue. One would expect that for a typical problem b i and λ i might be drawn from uncorrelated distributions. In this case, large values of z i will be likely to correspond to small values of λ i. Since z 0 can only be drawn from the average distribution (no information is known at this point), the estimated λ i is more likely to be large, as the initialization will be far from the optimum. Intuitively, the gradient is large because the optimum is far from the initialization, but the algorithm mistakes this large gradient for a large eigenvalue. On the other hand, when the parameter is initialized close to its optimum, the algorithm will mistakenly believe the eigenvalue is small, and so take relatively large steps. Although they do not affect the overall convergence much on their own (since the parameter is near its optimum), these steps can add significant noise to the process, making it difficult to accurately measure gradients and therefore find the optimum in other parameters. This problem will be significantly worse for Adam, which forgets its initialization with some decay factor β 2. In that case, as each parameter reaches its optimum, its estimated eigenvalue λ i drops and the step size gets correspondingly increased. In fact, the overall algorithm can be divergent as each parameter reaches its optimum, as the step size will grow unbounded unless α is scheduled to decline properly or a bounding method like AMSGrad is used . In addition, reviewing our earlier assumption of small, these algorithms will perform worse for small eigenvalues λ i < /α. This might be especially bad in Adam where late in training when the We finally note that the Hessians in deep learning problems are not diagonal . As such, each element might be better optimized by a learning rate that better serves both its min and max eigenvalues. Overall, this understanding has led us to believe that adaptive moments might effectively estimate λ i when it is large, but might be less effective when it is small. In order to incorporate this information about large eigenvalues, as well as optimize the learning rate to account for variation in the eigenvalues contributing to convergence of a particular component, we consider the an update to Eq. 1, whereλ is an average eigenvalue and η is a new hyper-parameter that controls the weighting of the eigenvalue estimation. Here we have addedλ to the numerator so that α does not need to absorb the changes to the RMS error as it does in Eq. 4. This also recovers the SGD convergence guarantees, since the step is always within a factor of η to an SGD step. In addition, this will allow us to recover SGD with momentum exactly in the limit η → 0. We use the adaptive gradient estimation, wherev t is the mean value of v t, to write, One issue with the above estimation is that its variance is very large at the beginning of training . It was suggested that this is the reason that warmup is needed for Adam and shown Input: θ 0 ∈ F: initial parameters, {α t > 0} T t=1: learning rate, α wd, β 1, β 2, η,: other hyperparameters, J t (θ): loss function Output: Average over all elements Calculate the denominator Perform the update end for return θ T that rectifying it can make warmup unnecessary. Where v t is the average of n t elements and v ∞ the average of n ∞, we define r t = n t /n ∞ and: This finally forms the basis for our algorithm. Our algorithm differs from Adam in a few other ways. First, the biased gradient estimate is used rather than the unbiased one. This matches the SGD implementation of momentum, and also avoids magnifying the large variance of early gradient estimates: In addition, the second moment v t is calculated in an unbiased way using an effectiveβ 2 (t), or by abuse of notationβ 2t:β This has a negligable impact on the performance of the algorithm, but makes tracking the moment over time easier since it does not need to be un-biased later. We then calculate the ratio of the number of samples n t used to calculate the moment v t to the steady state number of samples in the average n ∞: We finally note that the weight decay should be calculated separately from this update as in AdamW . In order to test the ability of this algorithm to reach better optima, we performed testing on a variety of different deep learning problems. In these problems we keep η at the default value of 1. Because the algorithm is the same as SGDM if η = 0 and is comparable to Adam when η = −1, getting at least as good as those algorithms is just a matter of parameter tuning. These are intended to show that SoftAdam performs remarkably well with a common parameter choice. For the best performance, the hyper-parameter η and learning rate schedule α should be optimized for a particular problem. We trained a variety of networks: 1 AlexNet , VGG19 with batch normalization , ResNet-110 with bottleneck blocks , PreResNet-56 with bottleneck blocks , DenseNet-BC with L=100 and k=12 on the CIFAR-10 dataset using SGD, AdamW and SoftAdam. For each model and optimization method, the weight decay was varied over [1e-4,2e-4,5e-4,1e-3,2e-3,5e-3]. For AdamW the learning rate was varied over [1e-4,2e-4,5e-4,1e-3,2e-3]. For each optimizer and architecture and the best was chosen, and three runs with separate initializations were used go generate the final data. The learning schedule reduced the learning rate by a factor of 10 at 50% and 75% through the total number of epochs. The are summarized in Table 1. We find that SoftAdam equals or outperforms SGD in training classifiers on this dataset. Due to the larger optimal weight decay constant, SoftAdam achieves lower validation loss at a higher train loss than SGD. We trained a 3-layer LSTM with 1150 hidden units per layer on the Penn Treebank dataset in the same manner as. For SoftAdam the weight drop was increased from 0.5 to 0.6. Results for the average of three random initializations are shown in Figure 2 (a) and are summarized in Table 2. For these parameters, SoftAdam outperforms SGD significantly but does not quite achieve the same as Adam. Note that for this experiment we chose Adam instead of AdamW for comparison due to its superior performance. We also trained a transformer using the fairseq package by on the IWSLT'14 German to English dataset. Results for each method with optimized hyperparameters are summarized in Table 3. Note that no warmup is used for training SoftAdam, but warmup is used for AdamW and In this paper, we have motivated and demonstrated a new optimization algorithm that naturally unifies SGD and Adam. We have focused our empirical on the default hyper-parameter setting, η = 1, and predetermined learning schedules. With these parameters, the algorithm was shown to produce optimization that is better than or equal to SGD and Adam on image classification tasks. It also performed significantly better than SGD on language modeling tasks. Together with finding the optimal values for η, we expect a better understanding of the learning schedule to bring light to the way in which the adaptive gradient methods improve convergence. SoftAdam now also makes it possible to create a learning schedule on η, which may be another fruitful avenue of research, expanding on the work of. Better understanding of how adaptive gradients improve the convergence of practical machine learning models during training will enable larger models to be trained to more accurately in less time. This paper provides a useful intuition for how that occurs and provides a new algorithm that can be used to improve performance across a diverse set of problems. # S t a t e i n i t i a l i z a t i o n i f l e n (s t a t e) == 0: s t a t e [" s t e p "] = 0 # E x p o n e n t i a l moving a v e r a g e o f g r a d i e n t v a l u e s s t a t e [" e x p a v g "] = t o r c h. z e r o s l i k e (p . d a t a) # E x p o n e n t i a l moving a v e r a g e o f # s q u a r e d g r a d i e n t v a l u e s s t a t e [" e x p a v g s q "] = t o r c h. z e r o s l i k e (p . d a t a) e x p a v g, e x p a v g s q = (s t a t e [ " e x p a v g "], s t a t e [" e x p a v g s q "], ) b e t a 1, b e t a 2 = g r o u p [" b e t a s "] s t a t e [" s t e p "] += 1 b e t a 2 h a t = min (b e t a 2, 1 . 0 − 1 . 0 / ( s t a t e [ " s t e p "] ) ) r b e t a = (1 − b e t a 2) / (1 − b e t a 2 h a t) e t a h a t 2 = (g r o u p [ " e t a "] * g r o u p [" e t a "] * r b e t a ) # Decay t h e f i r s t and s e c o n d moment w i t h t h e # r u n n i n g a v e r a g e c o e f f i c i e n t e x p a v g. mul (b e t a 1). a d d r e t u r n l o s s

An algorithm for unifying SGD and Adam and empirical study of its performance

For complex constraints in which it is not easy to estimate the gradient, we use the discounted penalty as a guiding signal. We prove that under certain assumptions it converges to a feasible solution.


Question: represent the scientific article to find a short tldr.

Question: The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. In this work, we discover the interaction between sub-tasks from their ing state-action trajectory sequences using a directed graphical model. We propose a new algorithm based on the generative adversarial imitation learning framework which automatically learns sub-task policies from unsegmented demonstrations. Our approach maximizes the directed information flow in the graphical model between sub-task latent variables and their generated trajectories. We also show how our approach connects with the existing Options framework, which is commonly used to learn hierarchical policies. Complex human activities can often be broken down into various simpler sub-activities or sub-tasks that can serve as the basic building blocks for completing a variety of complicated tasks. For instance, when driving a car, a driver may perform several simpler sub-tasks such as driving straight in a lane, changing lanes, executing a turn and braking, in different orders and for varying times depending on the source, destination, traffic conditions etc. Using imitation learning to learn a single monolithic policy to represent a structured activity can be challenging as it does not make explicit the sub-structure between the parts within the activity. In this work, we develop an imitation learning framework that can learn a policy for each of these sub-tasks given unsegmented activity demonstrations and also learn a macro-policy which dictates switching from one sub-task policy to another. Learning sub-task specific policies has the benefit of shared learning. Each such sub-task policy also needs to specialize over a restricted state space, thus making the learning problem easier. Previous works in imitation learning BID16 BID7 focus on learning each sub-task specific policy using segmented expert demonstrations by modeling the variability in each sub-task policy using a latent variable. This latent variable is inferred by enforcing high mutual information between the latent variable and expert demonstrations. This information theoretic perspective is equivalent to the graphical model shown in FIG0 (Left), where the node c represents the latent variable. However, since learning sub-task policies requires isolated demonstrations for each sub-task, this setup is difficult to scale to many real world scenarios where providing such segmented trajectories is cumbersome. Further, this setup does not learn a macro-policy to combine the learned sub-task policies in meaningful ways to achieve different tasks. In our work, we aim to learn each sub-task policy directly from unsegmented activity demonstrations. For example, given a task consisting of three sub-tasks -A, B and C, we wish to learn a policy to complete sub-task A, learn when to transition from A to B, finish sub-task B and so on. To achieve this we use a causal graphical model, which can be represented as a Dynamic Bayesian Network as. Right: Causal model in this work. The latent code causes the policy to produce a trajectory. The current trajectory, and latent code produce the next latent code shown in FIG0 (Right). The nodes c t denote latent variables which indicate the currently active sub-task and the nodes τ t denote the state-action pair at time t. We consider as given, a set of expert demonstrations, each of which is represented by τ = {τ 1, · · ·, τ T} and has a corresponding sequence of latent factors c = {c 1, · · ·, c T −1}. The sub-activity at time t dictates what state-action pair was generated at time t. The previous sub-task and the current state together cause the selection of the next sub-task. As we will discuss in Section 3, extending the use of mutual information to learn sub-task policies from unsegmented demonstrations is problematic, as it requires learning the macro-policy as a conditional probability distribution which depends on the unobserved future. This unobserved future is unknown during earlier points of interaction (FIG0). To alleviate this, in our work we aim to force the policy to generate trajectories that maximize the directed information or causal information BID17 flow from trajectories to latent factors of variation within the trajectories instead of mutual information. Using directed information requires us to learn a causally conditioned probability distribution BID12 which depends only on the observed past while allowing the unobserved future to be sequentially revealed. Further, since there exists feedback in our causal graphical model i.e., information flows from the latent variables to trajectories and vice versa, directed information also provides a better upper bound on this information flow between the latent variables and expert trajectories than does the conventional mutual information BID17 BID12.We also draw connections with existing work on learning sub-task policies using imitation learning with the options framework BID27 BID3. We show that our work, while derived using the information theoretic perspective of maximizing directed information, bears a close resemblance to applying the options framework in a generative adversarial imitation setting. Thus, our approach combines the benefits of learning hierarchical policies using the options framework with the robustness of generative adversarial imitation learning, helping overcome problems such as compounding errors that plague behaviour cloning. In summary, the main contributions of our work include:• We extend existing generative adversarial imitation learning frameworks to allow for learning of sub-task specific policies by maximizing directed information in a causal graph of subactivity latent variables and observed trajectory variables.• We draw connections between previous works on imitation learning with sub-task policies using options and show that our proposed approach can also be seen as option learning in a generative adversarial setting.• We show through experiments on both discrete and continuous state-action spaces, the ability of our approach to segment expert demonstrations into meaningful sub-tasks and combine sub-task specific policies to perform the desired task.2 RELATED WORK Imitation Learning BID22 aims at learning policies that can mimic expert behaviours from demonstrations. Modeling the problem as a Markov Decision Process (MDP), the goal in imitation learning is to learn a policy π(a|s), which defines the conditional distribution over actions a ∈ A given the state s ∈ S, from state-action trajectories τ = (s 0, a 0, · · ·, s T) of expert behaviour. Recently, BID8 introduced an imitation learning framework called Generative Adversarial Imitation Learning (GAIL) that is able to learn policies for complex high-dimensional physics-based control tasks. They reduce the imitation learning problem into an adversarial learning framework, for which they utilize Generative Adversarial Networks (GAN) BID5. The generator network of the GAN represents the agent's policy π while the discriminator network serves as a local reward function and learns to differentiate between state-action pairs from the expert policy π E and from the agent's policy π. Mathematically, it is equivalent to optimizing the following, DISPLAYFORM0 InfoGAIL BID16 and BID7 solve the problem of learning from policies generated by a mixture of experts. They introduce a latent variable c into the policy function π(a|s, c) to separate different type of behaviours present in the demonstration. To incentivize the network to use the latent variable, they utilize an information-theoretic regularization enforcing that there should be high mutual information between c and the state-action pairs in the generated trajectory, a concept that was first introduced in InfoGAN BID2. They introduce a variational lower bound L 1 (π, Q) of the mutual information I(c; τ) to the loss function in GAIL. DISPLAYFORM1 The modified objective can then be given as, DISPLAYFORM2 InfoGAIL models variations between different trajectories as the latent codes correspond to trajectories coming from different demonstrators. In contrast, we aim to model intra-trajectory variations and latent codes in our work correspond to sub-tasks (variations) within a demonstration. In Section 3, we discuss why using a mutual information based loss is infeasible in our problem setting and describe our proposed approach. Consider an MDP with states s ∈ S and actions a ∈ A. Under the options framework BID27, an option, indexed by o ∈ O consists of a sub-policy π(a|s, o), a termination policy π(b|s,ō) and an option activation policy π(o|s). After an option is initiated, actions are generated by the sub-policy until the option is terminated and a new option is selected. Options framework has been studied widely in RL literature. A challenging problem related to the options framework is to automatically infer options without supervision. Option discovery approaches often aim to find bottleneck states, i.e., states that the agent has to pass through to reach the goal. Many different approaches such as multiple-instance learning BID18, graph based algorithms BID19 BID26 have been used to find such bottleneck states. Once the bottleneck states are discovered, the above approaches find options policies to reach each such state. In contrast, we propose a unified framework using a information-theoretic approach to automatically discover relevant option policies without the need to discover bottleneck states. BID3 formulate the options framework as a probabilistic graphical model where options are treated as latent variables which are then learned from expert data. The option policies (π(a|s, o)) are analogous to sub-task policies in our work. These option policies are then learned by maximizing a lower bound using the Expectation-Maximization algorithm BID20. We show how this lower bound is closely related to the objective derived in our work. We further show how this connection allows our method to be seen as a generative adversarial variant of their approach. propose to extend the EM based approach to multiple levels of option hierarchies. Further work on discovery of deep continuous options allows the option policy to also select a continuous action in states where none of the options are applicable. Our proposed approach can also be extended to multi-level hierarchies (e.g. by learning VAEs introduced in section 3 with multiple sampling layers) or hybrid categorical-continuous macro-policies (e.g. using both categorical and continuous hidden units in the sampling layer in VAE). BID25 learn options by assuming knowledge of task sketches BID0 along with the demonstrations. The work proposes a behavior cloning based approach using connectionist temporal classification BID6 to simultaneously maximize the joint likelihood of the sketch sequences and the sub-policies. Our proposed approach does not expect task sketches as input, making it more amenable to problems where labeling demonstrations with sketch labels is difficult. Prior work in robot learning has also looked at learning motion primitives from unsegmented demonstrations. These primitives usually correspond to a particular skill and are analogous to options. BID21 used the Beta-Process Autoregressive Hidden Markov Model (BP-AR-HMM) to segment expert demonstrations and post-process these segments to learn motion primitives which provide the ability to use reinforcement learning for policy improvement. Alternately, BID14 use Dirichlet Process Gaussian Mixture Model (DP-GMM) to segment the expert demonstrations by finding transition states between linear dynamical segments. Similarly, BID23 use the BP-AR-HMM framework to initially segment the expert demonstrations and then use an inverse reinforcement learning step to infer the reward function for each segment. The use of appropriate priors allows these methods to discover options without a priori knowledge of the total number of skills. BID15 model the task of manipulation as an autoregressive Hidden Markov Model where the hidden phases of manipulation are learned from data using EM. However, unlike the above methods, in our proposed approach we also learn an appropriate policy over the extracted options. We show how this allows us to compose the individual option policies to induce novel behaviours which were not present in the expert demonstrations. As mentioned in the previous section, while prior approaches can learn to disambiguate the multiple modalities in the demonstration of a sub-task and learn to imitate them, they cannot learn to imitate demonstrations of unsegmented long tasks that are formed by a combination of many small sub-tasks. To learn such sub-task policies from unsegmented deomonstrations we use the graphical model in FIG0 (Right), i.e., consider a set of expert demonstrations, each of which is represented by τ = {τ 1, · · ·, τ T} where τ t is the state-action pair observed at time t. Each such demonstration has a corresponding sequence of latent variables c = {c 1, · · ·, c T −1} which denote the sub-activity in the demonstration at any given time step. As noted before, previous approaches BID16 BID7 model the expert sub-task demonstrations using only a single latent variable. To enforce the model to use this latent variable, these approaches propose to maximize the mutual information between the demonstrated sequence of state-action pairs and the latent embedding of the nature of the sub-activity. This is achieved by adding a lower bound to the mutual information between the latent variables and expert demonstrations. This variational lower bound of the mutual information is then combined with the the adversarial loss for imitation learning proposed in BID8. Extending this to our setting, where we have a sequence of latent variables c, yields the following lower bound on the mutual information, DISPLAYFORM0 Observe that the dependence of q on the entire trajectory τ precludes the use of such a distribution at test time, where only the trajectory up to the current time is known. To overcome this limitation, in this work we propose to force the policy to generate trajectories that maximize the directed or causal information flow from trajectories to the sequence of latent sub-activity variables instead. As we show below, by using directed information instead of mutual information, we can replace the dependence on τ with a dependence on the trajectory generated up to current time t. The directed information flow from a sequence X to Y is given by, DISPLAYFORM1 where H(Y X) is the causally-conditioned entropy. Replacing X and Y with sequences τ and c, DISPLAYFORM2 Here DISPLAYFORM3 which uses an approximate posterior q(c t |c 1:t−1, τ 1:t) instead of the true posterior p(c t |c 1:t−1, τ 1:t) can then be derived to get (See Appendix A.1 for the complete derivation), DISPLAYFORM4 Thus, by maximizing directed information instead of mutual information, we can learn a posterior distribution over the next latent factor c given the latent factors discovered up to now and the trajectory followed up to now, thereby removing the dependence on the future trajectory. In practice, we do not consider the H(c) term. This gives us the following objective, DISPLAYFORM5 We call this approach Directed-Info GAIL. Notice that, to compute the loss in equation 3, we need to sample from the prior distribution p(c 1:t). In order to estimate this distribution, we first pre-train a variational auto-encoder (VAE) BID11 on the expert trajectories, the details of which are described in the next sub-section. Figure 2 (left) shows the design of the VAE pictorially. The VAE consists of two multi-layer perceptrons that serve as the encoder and the decoder. The encoder uses the current state s t and the previous latent variable c t−1 to produce the current latent variable c t. We used the Gumbel-softmax trick BID9 to obtain samples of latent variables from a categorical distribution. The decoder then takes s t and c t as input and outputs the action a t. We use the following objective, which maximizes the lower bound of the probability of the trajectories p(τ), to train our VAE, DISPLAYFORM0 Figure 2 (right) gives an overview of the complete method. The VAE pre-training step allows us to get approximate samples from the distribution p(c 1:t) to optimize equation 4. This is done by using q to obtain samples of latent variable sequence c by using its output on the expert demonstrations. In practice, we fix the weights of the network q to those obtained from the VAE pre-training step when optimizing the Directed-Info GAIL loss in equation 4. In BID3 the authors provide a probabilistic perspective of the options framework. Although, BID3 Figure 2: Left: VAE pre-training step. The VAE encoder uses the current state (s t), and previous latent variable (c t−1) to produce the current latent variable (c t). The decoder reconstructs the action (a t) using s t and c t. Right: An overview of the proposed approach. We use the VAE pre-training step to learn an approximate prior over the latent variables and use this to learn sub-task policies in the proposed Directed-Info GAIL step. DISPLAYFORM0 Note that the first term in equation 6 i.e., the expectation over the distribution log p(c t |s t, c t−1) is the same as equation 3 of our proposed approach with a one-step Markov assumption and a conditional expectation with given expert trajectories instead of an expectation with generated trajectories. The second term in equation 6 i.e., the expectation over log π(a t |s t, c t) is replaced by the GAIL loss in equation 4. Our proposed Directed-Info GAIL can be therefore be considered as the generative adversarial variant of imitation learning using the options framework. The VAE behaviour cloning pretraining step in equation 5 is exactly equivalent to equation 6, where we use approximate variational inference using VAEs instead of EM. Thus, our approach combines the benefits of both behavior cloning and generative adversarial imitation learning. Using GAIL enables learning of robust policies that do not suffer from the problem of compounding errors. At the same time, conditioning GAIL on latent codes learned from the behavior cloning step prevents the issue of mode collapse in GANs. We present on both discrete and continuous state-action environments. In both of these settings we show that our method is able to segment out sub-tasks from given expert trajectories, learn sub-task conditioned policies, and learn to combine these sub-task policies in order to achieve the task objective. Table 1: A comparison of returns for continuous environments. The returns were computed using 300 episodes. Our approach gives comparable returns to using GAIL but also segments expert demonstrations into sub-tasks. The proposed Directed-Info GAIL approach improves over the policy learned from the VAE pre-training step. For the discrete setting, we choose a grid world environment which consists of a 15 × 11 grid with four rooms connected via corridors as shown in FIG1. The agent spawns at a random location in the grid and its goal is to reach an apple, which spawns in one of the four rooms randomly, using the shortest possible path. Through this experiment we aim to see whether our proposed approach is able to infer sub-tasks which correspond to meaningful navigation strategies and combine them to plan paths to different goal states. FIG1 shows sub-task policies learned by our approach in this task. The two plots on the left correspond to two of the four different values of the latent variable. The arrow at every state in the grid shows the agent action (direction) with the highest probability in that state for that latent variable. In the discussion that follows, we label the rooms from 1 to 4 starting from the room at the top left and moving in the clockwise direction. We observe that the sub-tasks extracted by our approach represent semantically meaningful navigation plans. Also, each latent variable is utilized for a different sub-task. For instance, the agent uses the latent code in FIG1 (a), to perform the sub-task of moving from room 1 to room 3 and from room 2 to room 4 and the code in FIG1 (b) to move in the opposite direction. Further, our approach learns to successfully combine these navigation strategies to achieve the given objectives. For example, FIG1 show examples of how the macro-policy switches between various latent codes to achieve the desired goals of reaching the apples in rooms 1 and 2 respectively. To validate our proposed approach on continuous control tasks we experiment with 5 continuous state-action environments. The first environment involves learning to draw circles on a 2D plane and is called Circle-World. In this experiment, the agent must learn to draw a circle in both clockwise and counter-clockwise direction. The agent always starts at, completes a circle in clockwise direction and then retraces its path in the counter-clockwise direction. The trajectories differ in the radii of the circles. The state s ∈ R 2 is the (x,y) co-ordinate and the actions a ∈ R 2 is a unit vector representing the direction of motion. Notice that in Circle-World, the expert trajectories include two different actions (for clockwise and anti-clockwise direction) for every state (x, y) in the trajectory, thus making the problem multi-modal in nature. This requires the agent to appropriately disambiguate between the two different phases of the trajectory. Further, to show the scalability of our approach to higher dimensional continuous control tasks we also show experiments on Pendulum, Inverted Pendulum, Hopper and Walker environments, provided in OpenAI Gym BID1. Each task is progressively more challenging, with a larger state and action space. Our aim with these experiments is to see whether our approach can identify certain action primitives which helps the agent to complete the given task successfully. To verify the effectiveness of our proposed approach we do a comparative analysis of our with both GAIL BID8 and the supervised behavior cloning approaching using a VAE. To generate expert trajectories we train an agent using Proximal Policy Optimization BID24. We used 25 expert trajectories for the Pendulum and Inverted Pendulum tasks and 50 expert trajectories for experiments with the Hopper and Walker environments. Figures 4(a, b, c) show on the Circle-World environment. As can be seen in Figure 4(a, b), when using two sub-task latent variables, our method learns to segment the demonstrations into two intuitive sub-tasks of drawing circles in clockwise and counterclockwise directions. Hence, our method is able to identify the underlying modes and thus find meaningful sub-task segmentations from unsegmented data. We also illustrate how the learned sub-task policies can be composed to perform new types of behavior that were unobserved in the expert data. In Figure 4 (c) we show how the sub-task policies can be combined to draw the circles in inverted order of direction by swapping the learned macro-policy with a different desired policy. Thus, the sub-task policies can be utilized as a library of primitive actions which is a significant benefit over methods learning monolithic policies. We now discuss the on the classical Pendulum environment. Figure 4(d) shows the sub-task latent variables assigned by our approach to the various states. As can be seen in the figure, the network is able to associate different latent variables to different sub-tasks. For instance, states that have a high velocity are assigned a particular latent variable (shown in blue). Similarly, states that lie close to position 0 and have low velocity (i.e. the desired target position) get assigned another latent variable (shown in green). The remaining states get classified as a separate sub-task. Figure 5 shows the on the higher dimensional continuous control, Hopper and Walker, environments. Figure 5 (a) shows a plots for sub-task latent variable assignment obtained on these environments. Our proposed method identifies basic action primitives which are then chained together to effectively perform the two locomotion tasks. Figure 5(b) shows that our approach learns to assign separate latent variable values for different action primitives such as, jumping, mid-air and landing phases of these tasks, with the latent variable changing approximately periodically as the agent performs the periodic hopping/walking motion. Finally, in Table 1 we also show the quantitative evaluation on the above continuous control environments. We report the mean and standard deviations of the returns over 300 episodes. As can be seen, our approach improves the performance over the VAE pre-training step, overcoming the issue of compounding errors. The performance of our approach is comparable to the state-of-the-art GAIL BID8. Our method moreover, has the added advantage of segmenting the demonstrations into sub-tasks and also providing composable sub-task policies. Segmentations obtained using our proposed Directed-Info GAIL method on FetchPickandPlace-v1. Returns VAE −14.07 ± 5.57 GAIL −13.29 ± 5.84 Directed-Info GAIL −11.74 ± 5.87GAIL + L2 loss −12.05 ± 4.94 Directed-Info GAIL + L2 loss −9.47 ± 4.84 Table 2: Mean returns over 100 episodes on FetchPickandPlace-v1 environment, calculated using the'dense' reward setting. We further analyze our proposed approach in more detail in the Appendix. In Appendix A.4 we visualize the sub-tasks in a low-dimensional sub-space. Also, in Appendix A.5 we show when using a larger dimensional sub-task latent variable. A video of our on Hopper and Walker environments can be seen at https://sites.google.com/view/directedinfo-gail. We further performed experiments on the FetchPickandPlace-v1 task in OpenAI Gym. In each episode of this task, the object and goal locations are selected randomly. The robot then must first reach and pick the object, and then move it to the goal location. We trained agents using both our proposed Directed-Info GAIL and the baseline GAIL approaches. We used 500 expert demonstrations. While our method was able to learn to segment the expert demonstrations into the Pick and Place sub-tasks correctly, as can be seen in Figure 6 and the videos at https://sites.google.com/view/directedinfo-gail/home#h.p_ 4dsbuC5expkZ, neither our approach, nor GAIL was able to successfully complete the task. In our preliminary , we found that the robot, in both our proposed approach and GAIL, would reach the object but fail to grasp it despite repeated attempts. To the best of our knowledge, no other work has successfully trained GAIL on this task either. Our preliminary experiments suggested that stronger supervision may be necessary to teach the agent the subtle action of grasping. In order to provide this supervision, we additionally trained the policy to minimize the L2 distance between the policy action and the expert action on states in the expert demonstrations. At every training step, we compute the discriminator and policy (generator) gradient using the Directed-Info GAIL (or in the baseline, GAIL) loss using states and actions generated by the policy. Along with this gradient, we also sample a batch of states from the expert demonstrations and compute the policy gradient that minimizes the L2 loss between actions that the policy takes at these states and the actions taken by the expert. We weigh these two gradients to train the policy. Table 2 shows the returns computed over 100 episodes. Adding the L2 measure as an additional loss led to significant improvement. Our proposed approach Directed-Info GAIL + L2 loss outperforms the baselines. Moreover, we believe that this quantitative improvement does not reflect the true performance gain obtained using our method. The reward function is such that a correct grasp but incorrect movement (e.g. motion in the opposite direction or dropping of the object) is penalized more than a failed grasp. Thus, the reward function does not capture the extent to which the task was completed. Qualitatively, we observed a much more significant difference in performance between the proposed approach and the baseline. This can be seen in the sample videos of the success and failure cases for our and the baseline method at https://sites.google.com/view/ directedinfo-gail/home#h.p_qM39qD8xQhJQ. Our proposed method succeeds much more often than the baseline method. The most common failure cases for our method include the agent picking up the object, but not reaching the goal state before the end of the episode, moving the object to an incorrect location or dropping the object while moving it to the goal. Agents trained using GAIL + L2 loss on the other hand often fail to grasp the object, either not closing the gripper or closing the gripper prematurely. We believe that our approach helps the agent alleviate this issue by providing it with the sub-task code, helping it disambiguate between the very similar states the agent observes just before and just after grasping. Learning separate sub-task policies can help improve the performance of imitation learning when the demonstrated task is complex and has a hierarchical structure. In this work, we present an algorithm that infers these latent sub-task policies directly from given unstructured and unlabelled expert demonstrations. We model the problem of imitation learning as a directed graph with sub-task latent variables and observed trajectory variables. We use the notion of directed information in a generative adversarial imitation learning framework to learn sub-task and macro policies. We further show theoretical connections with the options literature as used in hierarchical reinforcement and imitation learning. We evaluate our method on both discrete and continuous environments. Our experiments show that our method is able to segment the expert demonstrations into different sub-tasks, learn sub-task specific policies and also learn a macro-policy that can combines these sub-task. TAB3: Experiment settings for all the different environments for both DirectedInfo-GAIL and VAE-pretraining step respectively. Thus, by maximizing directed information instead of mutual information, we can learn a posterior distribution over the next latent factor c given the latent factors discovered up to now and the trajectory followed up to now, thereby removing the dependence on the future trajectory. In practice, we do not consider the H(c) term. This gives us the objective, DISPLAYFORM0 In practice, we fix q from the VAE pre-training and only minimize over the policy π in equation 4. BID24 to train our policy network with = 0.2. For the VAE pre-training step we set the VAE learning rate also to 3e −4. For the Gumbel-Softmax distribution we set an initial temperature τ = 5.0. The temperature is annealed using using an exponential decay with the following schedule τ = max(0.1, exp −kt), where k = 3e − 3 and t is the current epoch. In the Circle-World experiment, we added another loss term L s to VAE pre-training loss L V AE, which penalizes the number of times the latent variable switches from one value to another. FIG4 shows the segmentation of expert trajectories with and without the L s term. We observed that without adding the smoothing penalty, the VAE learns to segment the expert trajectories into semi-circles as shown in FIG4 (a). While a valid solution, this does not match with the intuitive segmentation of the task into two sub-tasks of drawing circles in clockwise and counter-clockwise directions. The smoothing term can be thought of as a prior, forcing the network to change the latent variable as few times as possible. This helps reach a solution where the network switches between latent variables only when required. FIG4 (b) shows an example of segmentation obtained on expert trajectories after smoothing. Thus, adding more terms to the VAE pre-training loss can be a good way to introduce priors and bias solutions towards those that match with human notion of sub-tasks. DISPLAYFORM0 In Figure 8, we show the plots expert states, reduced in dimensionality using Principal Component Analysis (PCA), in Hopper and Walker environments. States are color coded by the latent code assigned at these states. We reduced the dimension of states in Hopper from 11 to 2 and in Walker from 17 to 3. These low dimensional representations are able to cover ∼ 90% of variance in the states. As can be seen in the figure, states in different parts of the space get assigned different latent variables. This further shows that our proposed approach is able to segment trajectories in such a way so that states that are similar to each other get assigned to the same segment (latent variable). For the following discussion we will represent a k-dimensional categorical variable as belonging to ∆ k−1 simplex. To observe how the dimensionality of the sub-task latent variable affects our proposed approach we show with larger dimensionality for the categorical latent variable c t. Since DirectedInfo-GAIL infers the sub-tasks in an unsupervised manner, we expect our approach to output meaningful sub-tasks irrespective of the dimensionality of c t. Figure 9 shows for using a higher dimensional sub-task latent variable. Precisely, we assume c t to be a 8-dimensional one hot vector, i.e., c t ∈ ∆ 7.As seen in the above figure, even with a larger context our approach identifies similar basic action primitives as done previously when c t ∈ ∆ 3. This shows that despite larger dimensionality our approach is able to reuse appropriate context inferred previously. We also visualize the context values for the low-dimensional state-space embedding obtained by PCA. Although not perfectly identical, these context values are similar to the visualizations observed previously for c t ∈ ∆ 3. Thus our proposed approach is able, to some extent, infer appropriate sub-task representations independent of the dimensionality of the context variable.

represent the tldr for finding the corresponding scientific article

Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information

Represent the short summary for finding the corresponding scientific article (long).

DCEM learns latent domains for optimization problems and helps bridge the gap between model-based and model-free RL --- we create a differentiable controller and fine-tune parts of it with PPO


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The Convolutional Neural Network (CNN) has been successfully applied in many fields during recent decades; however it lacks the ability to utilize prior domain knowledge when dealing with many realistic problems. We present a framework called Geometric Operator Convolutional Neural Network (GO-CNN) that uses domain knowledge, wherein the kernel of the first convolutional layer is replaced with a kernel generated by a geometric operator function. This framework integrates many conventional geometric operators, which allows it to adapt to a diverse range of problems. Under certain conditions, we theoretically analyze the convergence and the bound of the generalization errors between GO-CNNs and common CNNs. Although the geometric operator convolution kernels have fewer trainable parameters than common convolution kernels, the experimental indicate that GO-CNN performs more accurately than common CNN on CIFAR-10/100. Furthermore, GO-CNN reduces dependence on the amount of training examples and enhances adversarial stability. Convolutional Neural Networks have been successfully applied in many fields during recent decades, but the theoretical understanding of the deep neural network is still in the preliminary stages. Although CNNs have strong expressive abilities, they have two clear deficiencies. First, as complex functional mappings, CNNs, like black boxes, cannot take full advantage of domain knowledge and prior information. Second, when little data is available for a certain task, CNNs' generalization ability weakens. This is due to overfitting, which may occur due to the large number of parameters and the large model size. Stemming from these two defects, a great deal of research has been done to modify CNNs BID7; ).Before CNNs were applied, traditional geometric operators had developed quite well. Each geometric operator represents the precipitation of domain knowledge and prior information. For example, the Sobel operator (Works) is a discrete difference operator, which can extract image edge information for edge detection. The Schmid operator is an isotropic circular operator, which extracts texture information from images for face recognition. The Histogram of Oriented Gradients (HOG) BID8 ) is a statistic operator of gradient direction, which extracts edge direction distributions from images for pedestrian detection and other uses. Many computer vision tasks require domain knowledge and prior information. For example, in BID2, the texture information from the image is used for an auxiliary diagnosis of a fracture. Geometric operators can make use of domain knowledge and prior information, but cannot automatically change parameter values by learning from data. Convolutional Neural Networks have strong data expression abilities and learning abilities, but they struggle to make use of domain knowledge. For better data learning, we have combined the two. It is natural to directly use geometric operators for pre-processing, and then classify the data through a Convolutional Neural Network . However, this method uses human experience to select geometric operator parameter values, and then carries out the Convolutional Neural Network learning separately. This method is a kind of two-stage technique, and without reducing parameter redundancy in a Convolutional Neural Network, it is difficult to achieve global optimization. The method proposed in this paper directly constructs geometric operator convolution and then integrates geometric operator convolution into a Convolutional Neural Network to form a new framework -the Geometric Operator Convolutional Neural Network. This method achieves global optimizations and utilizes the properties of geometric operators. In summary, the contributions of this work are as follows:• This framework can integrates many conventional geometric operators, which reveals its broad customization capabilities when handling diverse problems.• In theory, the same approximation accuracy and generalization error bounds are achieved when geometric operators meet certain conditions.• The Geometric Operator Convolutional Neural Network not only reduces the redundancy of the parameters, but also reduces the dependence on the amount of the training samples.• The Geometric Operator Convolutional Neural Network enhances adversarial stability. In recent years, Convolutional Neural Networks have been widely used in various classification and recognition applications BID19 BID15. Convolutional Neural Networks have achieved advanced success in various problems. All CNNs adopt an end-to-end approach to learning; however, each unique task is associated with its own distinctive domain knowledge and prior information. Thus, to improve classification accuracy, researchers use priori information that is tailored to each specific task and each specific Convolutional Neural Network. One way to do this is to use the traditional image processing algorithm as a preprocessing step. Another way is to use the traditional image processing algorithm to initialize convolution kernels. Classification accuracy is a primary concern for researchers in the machine-learning community. Different pre-processing models, such as filters or feature detectors, have been employed to improve the accuracy of CNNs. One example of this is the Gabor filter with CNN BID10. The Gabor filter is a feature extractor based on human vision. Besides the Gabor filter, some people also use Fisher vectors BID5, sparse filter Banks , and the HOG algorithm combined with a CNN to improve accuracy. Based on the human visual system, these filters are found to be remarkably well-suited for texture representation and discrimination. In the works by and , the Gabor filter is used to extract features from the input image in a pre-processing step. However, these methods require a kind of two-stage procedure that may not reach the optimal global solution. In addition, some scholars use traditional image processing algorithms to initialize convolutional kernels, such as building a Feature Pyramid Network with an image pyramid for multi-scale feature extraction . Geometric operators are widely used in traditional image processing algorithms. Many researchers use the Gabor filter to fix the first convolution layer, while other layers, which are common convolution layers, can be trained to improve their accuracy . John et al. simultaneously adopted the weight of the first layer convolution with the Gershgorin circle theorem and the Gabor filter constraint to improve the classification accuracy when Convolutional Neural Networks propagated backward. In BID1 and BID3, the authors have attempted to get rid of the pre-processing overhead by introducing Gabor filters in the first convolutional layer of a CNN. In addition, some researchers use filters to initialize multiple convolutional kernels. only used the Gabor function to create kernels in four directions to initialize the convolutional kernels from a Convolutional Neural Network. These methods change the initialization weight and use domain knowledge, but they do not reduce the redundancy of model parameters, and they do not enhance the transformation ability of the model. In this paper, a new network, the Geometric Operator Convolutional Neural Network, is proposed. This method integrates geometric operators, namely the filters, into a convolutional neural network. This network can not only make use of domain knowledge and prior information, but also reduce the redundancy of network parameters and enhance the ability of model transformation. This network's construction is described in detail in the following section. , the Roberts operator , the Laplace operator (van), the Gabor operator BID13, and so on. Each operator has different characteristics. Therefore, different geometric operators are used in different application scenarios, according to the characteristics of each unique problem. For example, SIFT looks for feature points in different scale spaces for pattern recognition and image matching. The Roberts operator uses local differences to find edges for edge detection, and the Laplace operator uses isotropic differentials to retain details for image enhancement. Geometric operators represent the precipitation of domain knowledge and prior knowledge. The GO-CNN is proposed in this paper, which uses the characteristics of geometric operators. The first step in this framework is to convolve geometric operators. In this paper, the Gabor operator and the Schmid operator are mainly used as examples to illustrate how to carry out convolutions and integrate these convolutions into CNNs. Other geometric operators in subsequent studies employ similar concepts. In order to study the frequency characteristics of local range signals, BID11 proposed the famous "Window" Fourier transform (also called the short-time Fourier transform, STFT) in the paper "Theory of communication" in 1946. This is now known as the Gabor operator; when combined with images, it is referred to as the Gabor filter. Until now, the Gabor filter has undergone many developments, and its primary characteristics are listed below. First, the Gabor filter has the advantages of both spatial and frequency signal processing. As shown in Eqn. 1.0, the Gabor operator is essentially a Fourier transform with a gaussian window. For an image, the window function determines its locality in the spatial domain, so the spatial domain information from different positions can be obtained by moving the center of the window. In addition, since the gaussian function remains the same after the Fourier transform, the Gabor filter can extract local information in the frequency domain. Second, the Gabor filter's response to biological visual cells may be an optimal feature extraction method. BID9 BID9 extended the Gabor function to a 2-dimensional form and constructed a 2D Gabor filter on this basis. It was surprising to find that the 2D Gabor filter was also able to maintain consistency with the mammalian model of retinal nerve cell reception. Third, the Gabor kernels are similar to the convolution kernels from the first convolutional layer in the CNN. From the visualization of the first convolutional layer in AlexNet, which was proposed by BID19. Some convolution kernels present geometric properties, as in the kernel function from the Gabor filter. From this feature, it can also be explained that there are parameter redundancies in the Convolutional Neural Network, and the Gabor operator can be convoluted and integrated into CNN. Lastly, the Gabor filter can extract directional correlation texture features from an image. DISPLAYFORM0 Since the Gabor operator combines with the CNN in the image, better feature expressions can be obtained. There are two main binding methods. First, the image is preprocessed by the Gabor operator, and then its features are extracted by the CNN. Next, the Gabor operator is convoluted to form a convolution layer, and then we integrate this convolution into the common Convolutional Neural Network. The second approach is used in this article. As shown in Eqn. 1.0, the Gabor kernel function has 5 parameters, which are obtained by learning and then regenerated into an m×m kernel. We replace the common convolution kernels with these Gabor kernels to form a convolutional layer. However, for the common convolutional layer, an m × m convolution kernel is generated by an identity mapping, which requires m 2 parameters. So, our method reduces the number of trainable parameters in the convolutional layer. In 2001, proposed a Gabor-like image filter, namely the Schmid operator. As shown in Eqn. 2.0, its composition is similar to the kernel function of the Gabor operator, so it retains the properties of the Gabor operator. In addition, the Schmid operator has rotation invariance. So, the Schmid operator is convoluted, and we integrate this convolution into common Convolutional Neural Network. This network improves the model's adversarial stability to rotation and improve the image feature extraction effect. Similar to the convolution of the Gabor operator, as shown in Eqn. 2.0, the Schmid kernel function has two parameters, which are obtained by learning and then generated by the Schmid kernel. Finally, we replace common convolution kernels with Schmid kernels to form a convolutional layer. DISPLAYFORM0 In this paper, only two geometric operator convolutions are explained. Similarly, for other geometric operators, operator kernels are generated by operator kernel functions, which replace common convolution kernels to form a convolutional layer. Due to the diversity of geometric operators, different geometric operators can be replaced with geometric operator convolutions, so the geometric operator convolution is customizable. There is a kind of geometric operator to form any kind of geometric operator convolution. Consequently, a question that must be addressed is how we combine multiple geometric operators with common CNNs to form GO-CNNs. Since the visualization of the first layer of convolution kernels maintains some geometric characteristics, we replace the convolution kernel in the first convolutional layer by kernel generated from geometric operators, and denote this kind of CNN as Geometric Operator Convolutional Neural Network (GO-CNN). In GO-CNN, kernels from the first convolutional layer are calculated by parameters of various geometric operator functions, and we call these operator functions as generator functions. Then, we concatenate all the calculated convolutional kernels in the last dimension to obtain a complete convolutional kernel. The full procedure is illustrated in FIG0. The generated convolution kernel is used as the weight of the first convolution layer in the Geometric Operator Convolutional Neural Network, and then the common convolution layer and output layer are connected. In this way, we have defined the forward propagation of the whole Geometric Operator Convolutional Neural Network. So, in backward propagation, the gradient of loss is transferred to the convolution kernel; this process is different from the usual convolution. Here, the convolution kernel generated by the geometric operator needs to further use the chain derivative rule (i. e., Eqn. 3.0, where L is the loss function, w is each convolution kernel, and p i is the parameter to generate each convolution kernel) to transfer the gradient to the parameters of each convolution kernel. Then, trainable parameters are updated by gradient descent algorithms, and the whole GO-CNN is complete. DISPLAYFORM0 The whole framework of the Geometric Operator Convolutional Neural Network has been introduced above. Next, we describe how to theoretically analyze the GO-CNN. It is theoretically proved that although the number of trainable parameters in the GO-CNN decreases, the effectiveness for computer vision tasks does not decrease. All detailed proofs are expanded in Appendix B. • We denote the input by DISPLAYFORM0, the corresponding label is {y i |y i = 0 or 1} DISPLAYFORM1 • The loss function is Mean Square Error.• The output of the neural network isỹ i for each input I i, and the empirical loss function is defined as follows:Ê DISPLAYFORM2 Definition 1 (Parametric Convolutional Kernel Space). Let f be a function that maps vector from R n to matrix in R m×m, n, m ∈ N +, and we call this function as convolution kernel generator function. Then we define Parametric Convolutional Kernel Space K f as: DISPLAYFORM3 We call n the parameter number, m the kernel size, od (short for output dimension) the output dimension. Since a convolutional kernel in a parametric convolutional kernel spaces is generated by function f, we call f as the generator function, and DISPLAYFORM4 Since kernel can be generated from a generator function by fewer parameters than ordinary kernel, the amount of trainable parameters of GO-CNN can be much smaller. However, reduction in parameters often causes loss of performance as the hypothesis space is smaller. In the simplest situation, we replace the ordinary kernel in the first convolutional layer by the parameter kernel generated from a parametric convolutional kernel space, and study on it. Definition 2 (GO-CNN). Assume that K f is a parametric convolutional kernel space. If the kernel in the first convolutional layer of a convolutional neural network is generated from K f, we call this network GO-CNN. We denote the set of GO-CNN by G f.GO-CNN is almost exactly the same as common CNN, except for the kernel in the first convolutional layer. We treat the first convolutional layer as a function from images to outputs, which then act as input of the following layer. If this function is not an injective function, meaning that different inputs can be mapped to identical outputs, then the network takes these identical outputs as the input of the following layers, meaning that the final outputs are still the same. However, the image inputs of the first convolutional layer are different, and corresponding labels can also be different. Thus, when the final outputs are the same, errors must occur. Therefore, we need to choose kernel carefully to make the function be an injective function. Since the convolution operator is a linear operator, we have the following proposition. Proposition 1. If the kernel of a convolutional layer, denoted by w, satisfies the following: I * w = 0 ⇔ I = 0, ∀I (6.0) where I is the layer input and * is the convolution operation, then this convolutional layer is an injective function. We find a necessary and sufficient condition for a convolutional layer to be an injective function. But which kernel satisfies this condition? In the proposition below, we show that 3 × 3 kernel generated by Gabor kernel function satisfies this condition. DISPLAYFORM5, where x = xcosθ + ysinθ, y = −xsinθ + ycosθ. Let K f be the corresponding parametric convolutional kernel space with kernel size m equal to 3 and sufficient output dimension od. Then, there exists kernel in K f satisfies the condition (6.0).As the kernel generated from K f could not meet the (6.0), we have the following definition: Definition 3 (Well-Defined GO-CNN). Let G ∈ G f, if there is a kernel generated by K f that satisfies (6.0), we call G a well-defined GO-CNN. We denote the set of all well-defined GO-CNNs as G * f. Corollary 1. If the generator function f is Gabor kernel function, the GO-CNN is well-defined. Now, let us consider a Convolutional Neural Network with one convolutional layer and two fully connected layers, and we will study the convergency of common CNN and GO-CNN. The detailed mathematical expression is expanded in Appendix B. Theorem 1. For any F ∈ F, where F is the set of common CNN, if the first fully connected layer is wide enough, the empirical loss of a well-defined GO-CNN can be that of common CNN controls. That is, for an arbitrary > 0, there exists d * ∈ N + and G ∈ G * f, such that when d 1 ≥ d *, the following inequality holds: DISPLAYFORM6 where F is the set of common CNN, if the first fully connected layer is wide enough, the generalization error of a well-defined GO-CNN can be that of common Convolutional Neural Network controlled. That is, for an arbitrary > 0, there exists d * ∈ N + and G ∈ G * f, such that when d 1 ≥ d *, the following inequality holds: DISPLAYFORM7 In Theorem. 2, we know that well defined GO-CNNs have almost the same generalization error as common CNNs. Therefore, we need to find which GO-CNNs are well defined. As GO-CNN with Gabor kernel function as the generator function is well defined, we have the following corollary. Corollary 2. Let f be Gabor kernel function, for any F ∈ F, if the first fully connected layer is wide enough, the generalization error GO-CNN G, which applies f as the generator function can be that of F controlled. That is, for an arbitrary > 0, there exists d DISPLAYFORM8 *, the following inequality holds: DISPLAYFORM9 More generally, if there are many generator functions in the first convolutional layer of a GO-CNN, when the number of kernels generated by Gabor kernel function is sufficient enough, this GO-CNN is also well defined. Therefore, we have the following corollary. Corollary 3. Let {f 1, f 2, · · ·, f T} be the set of generator functions. Suppose that there are od convolution kernels {k 1, k 2, · · ·, k od} in the first convolutional layer of a GO-CNN, denoted by G, and each k j is generated by function f tj, where 1 ≤ j ≤ od, 1 ≤ t j ≤ T. If there exists t * ∈ {1, 2, · · ·, T} such that f t * is Gabor kernel function, and the number of kernels generated by f t *, denoted by n t *, is sufficient big enough, then G is well defined, so that (8.0) holds. All experiments are performed on a single machine with CPU Intel Core i7-7700 CPU @ 3.60GHz × 8, GPU TITAN X (Pascal), and RAM 32G. Experimental details and more experiments are given in Appendix C, respectively. Theoretical analyses ensures that the GO-CNN has the same approximation accuracy and the same upper bound for generalization error as the common CNN. We verify this using two kinds of experiments on CIFAR-10/100 datasets. For the GO-CNN, the convolution kernels from the first layer are half trainable Schmid kernels and half trainable Gabor kernels. The basic network frameworks used in these experiments are ResNet18, ResNet34, and ResNet50 BID14.According to the cross-entropy curve of CIFAR-10/100 train sets, GO-CNN's value initially fell faster than the common CNN's, eventually almost reaching the same value. It is verified that GO-CNN achieves the same approximation accuracy as the common Convolutional Neural Network. According to the error rate curve of CIFAR-10/100 verification sets, the value of GO-CNN is lower than that of the common CNN. In addition, as shown in TAB1, the GO-CNN on the CIFAR-10 test set was 0.4% more accurate than the common CNN. On the CIFAR-100 test set, the GO-CNN was 0.5% more accurate than the common CNN. It is verified that the GO-CNN achieves the same generalization error bound as the common CNN. In many practical applications, such as the military, medical care, and so on, annotated data are often insufficient. Thus, a model's generalization ability for small data sets is of great importance. For CIFAR-10/100 and MNIST datasets, their train sets are large and their test sets are small. So, in these numerical experiments, the test set is directly used to train the model, and the train set is used to evaluate the model. For numerical experiments with CIFAR-10/100 datasets, the techniques and models used are the same as in Sec. 5.1. For numerical experiments with MNIST dataset, the basic network structure used in the experiment is LeNet . Similarly, in the GO-CNN, the first convolutional layer is replaced by the operator convolutional layer. The convolution kernels from the first layer are composed of trainable Gabor kernels and Schmid kernels. The other convolutional layers are the common convolutional layers. As shown in TAB2, from the perspective of the accuracy of MNIST and CIFAR-10/100, after the train set drops to one-fifth of the original train set, the accuracy of the common CNN falls faster than the GO-CNN. Moreover, the GO-CNN more accurate than the common CNN on the original train set. That is to say, the GO-CNN is better at predicting unknown data than the common CNN. The GO-CNN not only reduces the redundancy of the parameters, but also reduces the dependence on the amount of training samples. The current machine learning model, including the neural network and other models, is vulnerable to attacks from adversarial samples. In addition, CNNs show instability under attacks against adversarial samples BID12. So, it is very important to study the stability of adversarial It can be seen from TAB3 that when the test set is randomly rotated within 90 degrees, the difference of the GO-CNN is 1.21% lower than that of the common CNN. This verifies that the GO-CNN enhances the adversarial stability of rotated samples. When the small Gaussian disturbance (the mean is 0, the standard deviation is 0.3) is applied to the test set, the difference of the GO-CNN is 0.6% lower than that of the common CNN. This indicates that the GO-CNN enhances the adversarial stability of Gaussian disturbance adversarial samples. In sum, the GO-CNN enhances the adversarial stability of certain adversarial samples. In the above experiments, the Geometric Operator Convolutional Neural Network uses a priori knowledge from the field of medicine and provides a better recognition effect. Experiments about intelligent medical diagnoses of bone fractures is given in Appendix C. Although the trainable parameters decrease, GO-CNN still reaches the same approximation accuracy and a slightly lower generalization error upper bound when compared with the common CNN. And the GO-CNN reduces the dependence on training samples and enhances the adversarial stability of certain adversarial samples. In this paper, we present a novel framework named the Geometric Operator Convolution Neural Network, where the kernel in the first convolutional layer is replaced with kernels generated by geometric operator functions. This new network boasts several contributions. Firstly, the GO-CNN is customizable for diverse situations. Secondly, there is a theoretical guarantee in the learning framework of the GO-CNN. Thirdly, the GO-CNN reduces the dependence on training samples. Lastly, the GO-CNN enhances adversarial stability. In the future, we can explore a more appropriate geometric operator convolution block. A supplementary explanation about the Gabor and Schmid operators. The Gabor kernels are similar to the convolution kernels from the first convolutional layer in the CNN. An illustration of this similarity is shown in FIG1. In addition, the Gabor filter can extract directional correlation texture features from an image. As shown in FIG2, there are 40 Gabor As shown in FIG3, when the original image and a version of that image that has been rotated 90 degrees are both convolved with the same Schmid kernel, the ing characteristic graph exhibits only 90 degrees of rotation. So, the Schmid operator has rotation invariance. FIG1 ) let F and G be two hypothesis classes and let a ∈ R be a constant, we have: FIG0. With probability at least 1 − δ over the choice of S, the following holds for all h ∈ F: DISPLAYFORM0 DISPLAYFORM1 Proof of Proposition1.Assume that the proposition is not true, then there exist I 1 = I 2, such that I 1 * w = I 2 * w. Thus, if we set I = I 1 − I 2, we have I * w = (I 1 − I 2) * w = 0, since * is a linear operator, which means that I = 0 according to the condition. Therefore, the assumption is not true, and the is proved. Assume that there exists I ∈ R 3×3, I = 0, such that I * k = 0 holds for ∀k ∈ K f.We write I in the following matrix way: We define the pixel generator function f ij to be f x−1,y−1 (θ, σ, γ, λ, ψ). Then, we have the following equivalence: DISPLAYFORM0 DISPLAYFORM1 We will choose a variety of different parameters to discuss. DISPLAYFORM2 Since θ = 0, we have x = x, y = y, and the following: DISPLAYFORM3 We make the following shorthands for conveniency: DISPLAYFORM4 From Eqn.13.2, we can get: DISPLAYFORM5 The equation above means that, DISPLAYFORM6 Differentiate on both sides of parameter γ and get: DISPLAYFORM7 Since Eqn.13.7 holds for ∀γ, σ, which indicates that: DISPLAYFORM8 In the same way, we can get the following equation from Eqn.13.8: DISPLAYFORM9 Therefore, we have the following equations: DISPLAYFORM10 a 00 + a 02 + a 20 + a 22 = 0 a 01 + a 21 = 0 a 10 + a 12 = 0 a 11 = 0 (13.10) DISPLAYFORM11 In the same way, we have the following equations: DISPLAYFORM12 And we can get:(a 00 + a 02)h 1 + a 01 h 2 + (a 10 + a 12)h 3 + a 11 = 0, (13.12) which indicates that:a 00 + a 02 = 0 a 01 = 0 (13.13) From Eqn.13.10, we can get:a 00 + a 02 = 0 a 01 = a 21 = 0 (13.14) DISPLAYFORM13 We can get the following equations in the way just the same as discussed in situation II: DISPLAYFORM14 IV. θ = π/2, λ = 3, ψ = ±π/3.We have x = y, y = x this time, and we can get the following equations as the way discussed in situation II III,: DISPLAYFORM15 Combine equations 13.14 13.15 13.16, we can get: DISPLAYFORM16 We have x = √ 2 2 (x + y), y = √ 2 2 (y − x) and the following: DISPLAYFORM17 Therefore, we have DISPLAYFORM18 Combine equations 13.10 13.17 13.19, we can find that a ij = 0, f ori, j = 0, 1, 2, which means that I = 0. Therefore, the assumption that I = 0 is not true. For an arbitrary sized input I, we can focus on the 3 × 3 sized submatrix that will do inner product with the convolution kernel and get the same . Proof of Corollary1. From Prop.2, the is obvious. For the common CNN, denoted by F, we define the convolution kernel as k F. The weights of the rest of fully connected layers are {a F,1, a F,2}, and the biases of three layers are {b F,0, b F,1, b F,2}. Let σ stand for sigmoid activation function, then the convolutional layer C F and the fully connected layer F C F can be defined as follows: DISPLAYFORM19 Then, the last two fully connected layers can be defined as: DISPLAYFORM20 Therefore, the output before activation, denoted by F (x), and after activation, denoted byF (x), are defined as: DISPLAYFORM21 We denote the set of common CNN as F, that is, F = {F}, and the output before activation and after activation of input I i as F i,F i.For a GO-CNN G, we similarly define the convolutional kernel to be k g, and the weights and biases DISPLAYFORM22 Then, we have the following shorthand when the input is x: DISPLAYFORM23 We denote the output before activation and after activation of input I i as G i,G i as well. We maintain the same neuron number for each corresponding layer in common CNN and GO-CNN, that is to say, dim (b F,k DISPLAYFORM24, since the approximation ability is different when the neuron number is different. We define the width of each layer as DISPLAYFORM25 For the common CNN, denoted by F, we define the convolution kernel as k F . The weights of the rest of fully connected layers are {a F,1, a F,2}, and the biases of three layers are {b F,0, b F,1, b F,2}. Let σ stand for sigmoid activation function, then the convolutional layer C F and the fully connected layer F C F can be defined as follows: DISPLAYFORM26 Then, the last two fully connected layers can be defined as: DISPLAYFORM27 Therefore, the output before activation, denoted by F (x), and after activation, denoted byF (x), are defined as: DISPLAYFORM28 We denote the set of common CNN as F, that is, F = {F}, and the output before activation and after activation of input I i as F i,F i.For a GO-CNN G, we similarly define the convolutional kernel to be k g, and the weights and biases are {a G,1, a G,2, b G,0, b G,1, b G,2}. Then, we have the following shorthand when the input is x: DISPLAYFORM29 We denote the output before activation and after activation of input I i as G i,G i as well. We maintain the same neuron number for each corresponding layer in common CNN and GO-CNN, that is to say, dim (b F,k DISPLAYFORM30, since the approximation ability is different when the neuron number is different. We define the width of each layer as DISPLAYFORM31 Proof of Theorem1. Notice that DISPLAYFORM32 Apply absolute value on both sides DISPLAYFORM33 The last inequality holds as DISPLAYFORM34 We can fix parameters of ordinary CNN, so that there is a mapping between input I i and outputF i, and the mapping function isF as we have defined. We can also fix parameters of C G, and choose the convolution kernel of C G that satisfies (??) since G is a well-defined GO-CNN, so that C G is an injective function, which means that C −1 G exists. In the same time, D G = F C G,2 • σ • F C G,1 can be treated as a one hidden layer neural network. Combine with (16.2), we can get DISPLAYFORM0 Proof of Theorem2. From Theorem.1, we know that G satisfies the following inequality: DISPLAYFORM1 From Lemma.3, we know that DISPLAYFORM2 Since G * f ⊂ F, we have the following inequality from Lemma.2: DISPLAYFORM3 Combined with (17.2), we havê DISPLAYFORM4 The is proved! Proof of Corollary2. From Theorem.2 and Corollary.1, this is obvious. Let K be the set of k i such that the generator function of k i is f ti = f t * and denote the concatenation of all these k i ask. Suppose that there exists an input x, satisfies that x * k i = 0, i = 1, 2, · · ·, od, then x * k = 0, ∀k ∈ K. Therefore, x * k holds for any parameters. However, it is conflict with Prop.2.Therefore, the is proved! C APPENDIX Approximation accuracy and generalization error Recognizing objects in an actual scene is not dependent on corresponding domain knowledge but on humans' prior information. For object recognition tasks, the Geometric Operator Convolutional Neural Network's recognition effect is worth exploring. The commonly used public data sets for common object recognition are CIFAR-10 (ten categories) and CIFAR-100 (100 categories). They are all three-channel color images with a resolution of 32×32. The train set contains 50,000 images and the test set contains 10,000 images. The basic network frameworks used in these experiments are ResNet18, ResNet34, and ResNet50 BID14, which mainly consist of a new residual structure unit. In these experiments, four paddings are added on the four edges. Then, a random 32×32 cropping is performed, and a data enhancement method is carried out, which involves turning the image up and down. For both testing and training, the images' pixels are normalized to a 0-1 distribution. The Stochastic gradient descent optimization algorithm with 0.9 the momentum is used during the training process. The batch size is 100, the initial learning rate is 0.1, and the weight decay is 0.0005. The learning rate is reduced by one fifth per 60, 120, and 160 epochs. We report the performance of our algorithm on a test set after 200 epochs based on the average over five runs. As shown in FIG5, according to the cross-entropy curve of the CIFAR-10 and CIFAR-100 train sets, GO-CNN's value initially fell faster than the common CNN's, eventually almost reaching the same value. It is verified that Geometric Operator Convolutional Neural Network achieves the same approximation accuracy as the common Convolutional Neural Network. According to the error rate BID17 are generally used for visualization. The T-SNE visualization maps data points to a two-dimensional or three-dimensional probability distribution through affinitie transformation. Then, the data points are displayed with a two-dimensional or three-dimensional plane. In this paper, a two-dimensional T-SNE visualization is adopted to display the CIFAR-10 features extracted by the model. As shown in FIG6, the CIFAR-10 features extracted by the Geometric Operator Convolutional Neural Network are evenly separated from each other in the two-dimensional visualization of T-SNE, while the features extracted from the common Convolutional Neural Network are mixed. It is apparent that the features extracted by the GO-CNN are more separable; in other words, the features learned by the GO-CNN are more distinguishable and easy to classify with the last fully connected layer. Generalization MNIST is a public, handwritten recognition dataset with a total of ten classes. This dataset is a channel image with 28×28 resolution and a clean . The train set contains 50,000 images and the test set contains 10,000 images. For numerical experiments with the MNIST data set, the adaptive moment estimation BID18 optimization algorithm is used. In addition, as an image enhancement strategy, the image padding is increased to 32×32 during the training process. The batch size is 100, the initial learning rate is 0.001, and the weight decay is 0.0005. The learning rate stays the same until reaching 20,000 iterations. Consequently, we complete 20,000 iterations on one test set and average the performance over five runs in order to report the final performance evaluation of our algorithm. Application Medical images in China are developing rapidly, but specialist doctors are short of resources, and they are mainly concentrated in big cities and big hospitals. Many small and mediumsized cities do not have sufficient diagnostic imaging capacities, so many patients have to go to big cities in order to access better medical resources and obtain better treatment. Similarly, there are few Doctors usually judge whether a fracture has occurred based on whether there is a fracture line (texture) in the image. In BID2, the texture information from the image is used for an auxiliary diagnosis of a fracture. With prior information from the Schmid operator, we do preprocessing by Schmid operators to enhance the texture information from an image. Then, we use the CNN to conduct classification. However, the parameters of geometric operators in this two-stage method are preset by human experience. At this point, it is difficult for the local parameters obtained by the respective optimization to reach the global optimum. Thus, one may consider integrating the preprocessing of geometric operators into the deep network for global parameter learning without prior artificial empirical design parameters. In other words, this would mean using the GO-CNN proposed in this paper, wherein the convolution kernels from the first layer are all trainable Schmid kernels. Around 2,000 samples from X-rays taken at the Hainan People's Hospital were used as the data for the three kinds of intelligent fracture diagnosis models. Each sample was manually divided into bone The above three models are used for numerical experiments. The basic network framework used in the experiment is ResNet50 BID14, which mainly consists of a new residual structure unit. To balance the data during training, the number of fracture patches is increased to 4,016 by rotating the images and changing the of the images. In the test set, there are 145 fracture patches and 1,004 non-fracture patches. Then, five experiments are conducted to evaluate each model. The SGD algorithm and the finetune strategy are used during the training process, with a batch size of 50. The initial learning rate is 0.001 and the weight decay is 0.0005. The learning rate is reduced by one fifth every 4,000 iterations. Each data class is queued, and the data from each batch is averaged out of each data class during training. We report the performance of our algorithm on the test set after 12,000 iterations based on the average over five runs. According to TAB6, the Geometric Operator Convolutional Neural Network is the most accurate. Moreover, the fracture recall of the two-stage method is 0.77% higher than that of the Convolutional Neural Network, indicating that domain knowledge from the field of medicine is important for intelligent diagnosis. The fracture recall of the Geometric Operator Convolutional Neural Network is 2.21% higher than that of the two-stage method, which indicates that the Geometric Operator Convolutional Neural Network does make use of medical knowledge for fracture diagnosis. The integration of geometric operator into the deep neural network indeed achieve global optimization.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

Traditional image processing algorithms are combined with Convolutional Neural Networks，a new neural network.

We propose a SVD based method to explore the local dimension of activation manifold in deep neural networks.