Question: Represent the article for finding a single-sentence TLDR

Question: In this paper we present a novel optimization algorithm called Advanced Neuroevolution. The aim for this algorithm is to train deep neural networks, and eventually act as an alternative to Stochastic Gradient Descent (SGD) and its variants as needed. We evaluated our algorithm on the MNIST dataset, as well as on several global optimization problems such as the Ackley function. We find the algorithm performing relatively well for both cases, overtaking other global optimization algorithms such as Particle Swarm Optimization (PSO) and Evolution Strategies (ES). Gradient Descent (GD) and its variations like stochastic gradient descent BID2 are the de facto standard for training deep neural networks (DNNs) for tasks in various domains like Object Detection BID10, Robotic Grasping BID9 and Machine Translation BID1. Most of the field of Deep Learning is centered around algorithms similar to variants of Gradient Descent to find the optimal weights given desired input/output pairs BID7, BID4, BID14. However, there are also some limitations to using gradient-based optimization. For example, the neural network and the loss function have to be differentiable end-to-end. As a consequence, there are a number of problems that can not be directly modeled or solved without some alterations such as Formal Logic and Hard Attention BID11. Note that throughout this paper, we will refer to gradient-based methods collectively as SGD. Similarly, we will refer to Advanced Neuroevolution with the acronym AdvN.For those reasons, we developed a new algorithm which we call Advanced Neuroevolution. It is not a single algorithm, in truth. It is an ensemble of low-level algorithms, layered on top of each other. Those low-level algorithms have different scopes of operations addressing different levels of abstraction in the search process. For example, the perturbation mechanism addresses the introduction of noise into the models, the most basic operation. In contrast, the minimum distance mechanism addresses the global scale properties, i.e. the search regions. The goal is to traverse the search space as efficiently as possible without use of gradients. In the case of neural networks the search space is the weight space, including biases. Indeed, while this algorithm was developed primarily for training of deep neural networks, it can be used for other optimization tasks. In essence, we present the algorithm as an evolutionary optimization algorithm, with a focus on DNNs. There are many global optimization algorithms such as Evolution Strategies BID13, Particle Swarm Optimization BID8 and Simulated Annealing BID23. Each has its merits and limitations. Our aim is not to compete directly with those algorithms but rather to complement them and offer another option with its own merits and limitations. To evaluate the performance of such algorithms we can use well-known benchmark functions such as the Rastrigin or Ackley function. We recognize those functions and test Advanced Neuroevolution against them to assess its performance. In addition, there have been other approaches to using evolutionary optimization techniques to train DNNs, see and BID19 as recent examples. It reflects the awareness within the broader research community about the potential of such algorithms, and the need for alternatives to SGD. We don't see our algorithm replacing SGD, especially in fields where it is already quite successful such as Computer Vision. Our aim is to complement it, by offering another option. Furthermore, there is no reason why both can not be used in tandem as part of a grander learning strategy. We noted that there has been a limitation in learning potential due to the employment of SGD. Specifically, SGD is prone to losing the model's top performance throughout training iterations. This is usually due to dynamic elements in the environments, and sometimes can also arise from batch training. To elaborate, at one point in training t i the agent may achieve a certain high score, say 9000. At a later point in time, t i+1, there is no guarantee that the agent will not regress to a lower score when faced with the exact same conditions. While this behavior is a natural part of SGD, and happens frequently during the training progress, the algorithm eventually overcomes these temporary regressions. In Robotics, and associated fields such as Reinforcement Learning, this beahvior can amount to being sample-inefficient. That is, the agent usually takes a relatively large number of samples, possibly in the order of 10 6 to 10 8, to achieve competitive learning BID15. In such cases, a large amount of labeled data is required which is often costly to create. Alternatives like simulation have other potential issues with producing training data which is representative for reality. This is another reason why we wanted to investigate gradient-free approaches. The algorithm we propose always maintains its best performing agent. Indeed it is wholly different from SGD as an optimization step. Advanced Neuroevolution seeks to train neural networks based on the concept of evolving a pool of agents, whereas SGD iteratively optimizes a single agent. One advantage for this approach is that it inherently achieves exploration and exploitation, whereas SGD usually struggles on exploration due to its gradient-following constitution. We see this paper as a starting point, for the whole Robotics community to engage in developing of alternative optimization for DNNs used in Robotics and RL. Training neural networks using optimization algorithms is not a novel idea, see for example BID12. However, we aim to improve upon the shortcomings of other algorithms with the mechanisms we introduce here. Our algorithm should be able to handle sparse-rewards, exploration, exploitation, high-dimensionality, computational efficiency, pool-efficiency and sample-efficiency. The field of evolutionary computation is rich with spectacular implementations and algorithms. For example, BID18 use neuro-evolution as a proxy algorithm to evolve the architectures of the networks, rather than the networks themselves. They also use computer vision is a starter task, though they apply their method on the more challenging sets of CIFAR10 and CIFAR100. The networks themselves are trained using SGD. In contrast, we evolve the weights of the networks themselves rather than the architectures. Similarly, the work in BID3 ) also uses gradients to evolve the networks. The authors in train models of 4M parameters, and cite it as one of the largest models ever trained with evolutionary algorithms. Following this interesting path, we decided to use a model on a similar scale for the MNIST classification task, to see whether our algorithm can solve as large a network or not. We summarize the contributions of the paper as follows. 1) Introduce an evolutionary optimization algorithm called Advanced Neuroevolution (AdvN).2) Benchmark AdvN algorithm and other optimization algorithms, on solving global optimization functions.3) Test AdvN against functions with special properties such as the sparsely-rewarding Easom, to evaluate its behavior. For example, Easom is a function that has near-zero gradient everywhere except a small region compared to the search space. Gradient-following in this scenario would be an implausible strategy. 4) Test AdvN and SGD on hand-written digit classification dataset MNIST. We impose special conditions to see how the algorithm can adapt to typical training challenges. The algorithm is actually a set of smaller, locally-aware algorithms and functions all operating to achieve a global synergistic behavior. We describe each in turn as follows, and eventually integrate all of them so as to portray the entire algorithm. Please note that we prefer to use nomenclature that is different from traditional evolutionary optimization. It reflects our way of thinking about the problem. Also, using standard evolutionary/genetic nomenclature such as "Mutations", sometimes hinders the mental grasp on the problem because of the association with biological processes. We do not wish to crystallize around or closely subscribe to the mechanisms of Biological evolution. For those reasons, we developed our own notation, which we share in the proceeding sections. Another note is that we employ a fixed-size pool, and the fixed network structure. This is in contrast to other evolutionary algorithms where the network structure may be changed throughout the training process. For the first iteration of the algorithm the pool will consist of a set of random samples chosen chosen according to the weight initialization scheme, e.g. Xavier Normal BID6. The subsequent iterations of the pool will be made up of the following components: Elite, Anchors, Probes and Blends. The Elite is the historically best-performing sample. Anchors are the bestperforming N samples in a generation, where N is the desired number of anchors. These samples are copied over to the new pool as is. From each anchor M probes are spawned, where M is the number of probes desired. Those probes are clones of the anchor, each randomly perturbed. In addition to the anchors and probes, the Elite is copied over as well. The remaining slots are filled with blends. Blends can occur between an anchor and any other sample in the pool. The most basic step in the algorithm is the Perturbation step. This is equivalent to genetic mutation, prinicipally referring to the introduction of noise into the makeup of the system. It is the primary searching mechanism in the algorithm. Two important properties are the magnitude and shape of the noise distribution. We choose uniformly-distributed noise, centered at the origin. We refer to limits of the uniform distribution as Search Radius. It can be thought of as defining a virtual radius around the anchor where the probes will be cast in random directions. The greater the radius the farther away the probes will be casted. This tunes the explorative vs. exploitative properties of the algorithm. The limit of the uniform distribution, ie. Search Radius, is calculated with respect to the integrity value and 2 pre-defined constants. Search radius is calculated as DISPLAYFORM0 where p = (1-integrity), λ and lr are scalar constants. The learning rate, lr, scales the function, controlling the search radius. The shifted, scaled hyperbolic tangent function has attractive properties in the range. It has an almost-flat slope near 0 and an almost-flat slope near 1. This allows the algorithm to spend more time searching low-energy regions where the most likely rewards are, ie. exploitation. It also prevents the algorithm from searching exceedingly high-energy configurations, ie. controlling exploration. Similar to the Search Radius, is the number of selections. This variable determines how many weights will be perturbed. It is calculated as DISPLAYFORM1 where p = (1-integrity), and α and β are scalar constants. In the range this function starts with at the origin and increases until it saturates near top. This saturation limits the number of modifications in the network according to the chosen constants. Intuitively, making too many adjustments to a neural network model in one step is usually un-rewarding. In addition, when searching high-energy regions, ie. low integrity, the number of selections can exceed reasonable limits. This creates a lot of wasted cycles where the search is not profitable. The experimenter chooses the constants introduced in equations 1 and 2 before starting the algorithm. Blending mechanism takes randomly-chosen weights from 2 components to yield a blend. The first component is always one of the anchors, picked at random. The second component can be any sample in the pool. This is commonly referred to as Crossover. The first component is cloned, to form the basis. Then a number of weights are randomly picked from the second component. This number is calculated by equation 2, using the same constants. The importance of blending is that it potentially allows us to explore regions in the search space which are otherwise unreachable without effort. Intuitively, blends have the potential to transport samples farther away from the current search region, shortcutting the general mechansim of perturbations. Blends help to extend the actively searched area in the search space based on already known favorable samples. There are two sorts of elites, a generational elite, and a historical elite. The generational elite is the best-performing sample of the current pool. The historical elite is the best performing sample across all generations. In our algorithm the generational elite is one of the Anchors, and the historical elite is simply called the Elite. The integrity concept defines the scope and magnitude of perturbations. The lower the integrity, the greater the magnitude and the scope of perturbations introduced into the model. The magnitude and scope of perturbations follow a non-linear trajectory with the integrity. The governing functions are equations 1 and 2. Generally, neural networks are not updated in bulk. Rather, updates should come as small incremental changes to the network. For those reasons, intuitively, we saturate the magnitude and scope of perturbations and blends in order not to exceed thresholds. Those thresholds are parameters of the algorithm. They control how aggressive the perturbations and blends are allowed to get. Integrity decreases when the current generation of samples does not yield a score that is incrementally better than the previous best score. There is a parameter in the algorithm that defines the required minimum percentage change in the score, we call it Minimum Entropy. For example, in the global optimization scenario, we set the minimum acceptable percentage improvement, ie. Minimum Entropy, to 1%. Thus, if the current generation does not improve upon the previous generation's score by at least 1% then the integrity is decreased by a fixed step size. The experimenter determines the appropriate step size. Backtracking curbs the behavior of integrity. It can be thought of as a layer on top of integrity. If integrity keeps decreasing, without improvement in the reward signal, then backtracking interferes and resets integrity re-inserts the Elite. Essentially, as integrity decreases we search higher-energy regions of the search space. Eventually, the model becomes "hot" from applying high-magnitude perturbations, and the weights start exploding. To fix this problem, backtracking resets the integrity back to maximum, and inserts the historical elite in place of the generational elite. When the elite is inserted, and the probes are spawned with high integrity, essentially copying the elite with minor modifications, then the entire pool "cools down". This mechanism makes it possible to explore high-energy configurations more safely. It is somewhat reminiscent of the simulated annealing. Anchors constitute the main search mechanism. Essentially, anchors constitute the best Nperforming agents, in order. Thus for example if there are 5 anchors, those will be the 5 bestperforming agents. Anchors are updated each generation, as well as being carried over to the next generation in case no improvement was made. The best anchor is also the generational elite. We spawn probes as clones of the anchors, but with the perturbations applied to them. Intuitively, what this means is that probes are spawned to search the local neighborhoods of the anchors. By having multiple anchors we search multiple regions in tandem. By having multiple probes, we search multiple local neighborhoods within those regions. Distance is a metric that represents how far the anchors can be from each other. It is calculated in terms of magnitudes and positions of difference between the weights of the anchors. Essentially, it's inefficient for the anchors to be close to each other. It would be practically searching the same region. Thus a minimal distance is defined. For that reason we choose the Canberra distance as given below: DISPLAYFORM0 where x,y represent the two anchors. Similar to distance where we don't want the anchors to collapse to a single region, we also introduce the concept of radial expansion to achieve the same goal. Radial expansion adaptively adjusts the perturbations magnitude and scope. We do this in order to cast the samples far away from each other so that we can maintain a full roster of anchors. For example, if we have a roster of 5 anchors and we're only using 3, then it means that all the other anchors are not far enough according to the distance metric. Thus we lose 2 anchors, and operate only 3 anchors. The remainder of the slots do not necessarily go to waste, they are given to the blending algorithm. However, this is not the ideal situation, since we expect the algorithm to be use the number of anchors allocated. Therefore, the mechanism of radial expansion increases the search space by increasing the value of the parameters governing the perturbations magnitude. This in turn forces the probes to be casted farther from the anchors, thereby allowing a greater diversity (and consequently distance) within the pool. We find that this mechanism is significantly involved in the training of networks solving the MNIST task. Collectively, the algorithm operates on two steps. The first is a conditioning step that takes into account the current and previous states of the algorithm, and decides the integrity to use, who are the anchors and the elite for the upcoming generation. The conditioning step makes use of the Distance and Expansion mechanisms described above. The second step is the execution step, the actual formation of the pool through the perturbation and blending steps. This section defines the implementation details of the algorithm, as well as the benchmarking processes. Across the entire experiment set, we use only a pool size of 50 samples. This remarkably low number emphasizes the objectives we are trying to achieve with this algorithm. The algorithm is written and implemented in PyTorch framework BID16. The networks used are common architectures. On this end, however, we think there is a vast opportunity to develop network architectures that are tailored to this sort of algorithm. Thus far, the architectures used have been tailored for SGD featuring layers such as Dropout. In our experiments, we find that the performance is sensitive to the architectural choice. However, optimizing the architecture is beyond the scope of this work. We use 4 Nvidia Titan V GPUs for data parallelization. That is, the models for copied into each GPU and the inference is run by dividing the dataset into 4 chunks, and then the loss is aggregated. For all our experiments we use a pool size of 50, 4 anchors, and 8 probes per anchor. Our aim was to showcase the algorithm on a number of tasks which are relatively simple. We decided to benchmark our performance against that of typical global optimization algorithms. We use the Inspyre and PyBrain libraries implementions BID20, along with the default parameters. It is typical that any optimization algorithm is sensitive to the choice of parameters. However, tuning those parameters is beyond the scope of this paper. We test on a set of 7 well-defined functions. Each function is tested 5 times, and the average number of generations required to converge is presented. Note that we define convergence as reaching a value within 0.06 of the global optimum. For example, if the global optimum is 0, then when the algorithm reaches 0.06 or less, it terminates. We solve the global optimization problem as a regression problem. The architecture we used is a single hidden layer of size 128. The functions are all 2-dimensional, therefore the network accepts two real numbers as the (x,y) origin and outputs 2 real numbers as optimal (x,y). The value of the function at the predicted (x,y) is the network's cost, and the aim is to minimize that cost. The input coordinates to the network are picked from a uniform distribution for each run, but remains constant throughout that run. The uniform distribution's limits differ for each function. Specifically, they are [-5,5] for Ackley, [-5.2,5 .2] for Rastrigin, [-2,2] for Rosenbrock, [-500,500] for Schwefel, [-15,-5,-3,3] for Bukin, [-20,20] for Easom and [-512,512] for Eggholder. For fair comparison they are the same for all the algorithms in the benchmark. We run the algorithms for a maximum of 5000 generations. If the algorithm has not converged, it is assumed that it is stuck in a local minima which it can't escape. If this is the case, then we calculate this particular run as taking 5000 generations. If all the runs don't converge within 5000 generations, we note this with a "+" sign, e.g. 5000+.In addition to regular benchmark functions, we wanted to test the algorithm against some special functions with special properties. For example, the Easom function is flat throughout except one regions where it is depressed. A typical gradient-following algorithm such as SGD will struggle to solve this function. It reflects an aspect of the sparse-reward problem in domains such as RL. Those special functions are not used for benchmarking, however, because the benchmarking libraries do not carry their implementations. In this set of experiments, we run the algorithm against a simple CNN to solve the MNIST digit classification problem. The model we trained had 4 convolutional layers of size 32, 64, 128 and 128 respectively with stride 2 and max-pooling in between, and a single fully-connected layer of size 512. The activation used was the Parametric ReLu function in PyTorch. The model has 4.7M parameters. To speed up the computations we switched to using half-precision floats. Interestingly, in our experimentation we found that the algorithm does not need much data at all to train. For that reason we used only a randomly chosen subset of the MNIST dataset, of 2000 images. When we run validation, however, we run against the entire validation set of 10,000 images. This not only tests how well the model generalizes, but also how much it can learn from a limited-dataset. This problem is of relevance to domains with relatively small volume of annonated data, such as Medical Imaging BID0.The test is terminated once the algorithm achieves a loss of 0.15 or lower, or if the number of generations exceeded 5000. We assert this limit in order to prevent runs of diminishing returns. Our goal is not to achieve or exceed the state of the art in MNIST classification. The system would need to be tuned beyond the casual effort, and then the training would take a different, more aggressive approach. Our aim is to showcase how the algorithm performs in an appreciatively high-dimensional search space. As mentioned earlier, the algorithm is tested on a number of common global optimization functions. For comparison we test several other optimization algorithms. Namely, Evolution Strategies BID13 DISPLAYFORM0 and some of them can definitely outperform the vanilla implementation shown here. Despite this, the clearly suggest that AdvN is a strong competitor. Our algorithm performs well and solves all the benchmarking functions in a relatively low number of generations. In addition, the algorithm solves the special functions we give it. The Eggholder function is notorious for being difficult since the global optimium is in a corner, and the function is not convex. Therefore it is quite difficult for an algorithm that doesn't explore enough to ever find the global optimum, but our algorithm does, and with only a pool size of 50 sampels. Similarly, a gradient-following algorithm would be stuck in the Easom function where the gradient is near zero throughout, save for one small region. We see that our algorithm explores well, and converges upon the reward-bearing region. The Bukin function has a narrow ridge where the global minimum lies. If the exploration steps are too large, the algorithm will generally skip over the ridge. Again, we highlight that those were achieved with the same set of parameters across all the functions. The parameters of the algorithm are not changed to solve a particular function. Rather, it has the required agility to deal with the different conditions imposed by the different functions. It is a desirable feature because throughout learning, the problem space may impose varying constraints and dynamics. The algorithm needs to be agile enough to handle those conditions as they arise. A sample run of the training loss function against the number of inferences is shown in Figure 4.2. The algorithm takes 2333 generations to converge to a training loss of 0.15, averaged over 5 runs. This corresponds to a validation accuracy of 90%. Similarly SGD takes 320 generations (ie. batches of 50) to converge to the same training loss, and achieves also a validation accuracy of 90%. It must be noted that 2333 generations over 2000 images with a pool size of 50 means that our algorithm on average runs inference 233,300,000 times. That is orders of magnitude more than SGD. In our run, SGD takes around 16,000 inferences to converge to a similar training loss. Therefore, we see a clear trade-off in the number of inferences required, in favor of SGD. This may be the case because the gradient is smooth in this task, it is possible that in other tasks, the gradient-space is more challenging to navigate, or uninformative. With only 2000 images from the training set, the model was able to achieve a relatively acceptable accuracy. Note that we don't use shuffling of those 2000 images. The network sees the same images in the same order. It would be interesting to see what role layers such as Dropout would play if we train other networks using AdvN.It is clear that the algorithm can handle the complexity of the 4.7M parameter network. It suggests that we can use it to train deeper, wider networks, on more complex tasks. Furthermore, we see that half-precision is not a hindrance for the algorithm. This bodes well for future iterations since we train on Volta architecture GPUs, which use tensor cores to execute those operations. It would be interesting to see if we can achieve the same level with even lower precision floats, if any such become available in the future. We presented the Advanced Neuroevolution algorithm as an alternative optimization step to SGD to train neural networks. The work is motivated by some limitations we perceived in gradient-based methods, such as differentiability and sample-inefficiency. The algorithm is benchmarked against other optimization algorithms on typical optimization problems. It performed satisfactorily well, and improved upon all of them. For fairness, we noted that the implementation of the other algorithms may not be optimized, and they can arguably perform better. Next our algorithm is tested on the MNIST digit classification task. It achieved 90% accuracy on the entire validation set using only 2000 images from the training set. In all our experiments, halfprecision floats are used in order to decrease the time of the computations. The computations are done only on 4 Titan V GPUs instead of thousands of CPU cores as in other evolutionary algorithms papers. This makes training of neural networks with evolutionary algorithms more tractable in terms of resource requirements. Finally, while not presented in this work, preliminary tests of our algorithm on RL tasks have been promising. It solves the assigned problems, though it takes longer than other approaches. We aim to improve upon the algorithm and the strategies employed in order to achieve competitive on RL and Robotics tasks.

represent the short summary for finding the corresponding scientific article (long)!

To give you a sense - "We present a mixed media assembly tutorial authoring system that streamlines creation of videos, images, text and dynamic instructions in situ." should be close to "Augmented Reality (AR) can assist with physical tasks such as object assembly through the use of “situated instructions”. These instructions can be in the form of videos, pictures, text or guiding animations, where the most helpful media among these is highly dependent on both the user and the nature of the task. Our work supports the authoring of AR tutorials for assembly tasks with little overhead beyond simply performing the task itself. The presented system, AuthAR reduces the time and effort required to build interactive AR tutorials by automatically generating key components of the AR tutorial while the author is assembling the physical pieces. Further, the system guides authors through the process of adding videos, pictures, text and animations to the tutorial. This concurrent assembly and tutorial generation approach allows for authoring of portable tutorials that fit the preferences of different end users. Physical task guidance can be delivered via Augmented Reality (AR) since assembly often requires both hands and continuous attention to the task. Additionally, assembly tutorials have instructions directly associated with physical objects, so AR can reduce the need for excessive context switching between the instructions and the physical structure by projecting those instructions into the environment. These benefits have been demonstrated in fields such as Facilities Management, Maintenance, and Internet of Things (IoT) device management. Additionally, prior work in AR assembly guidance has shown that these benefits can translate to carrying out assembly tasks. While significant previous work has looked at the benefits of following tutorials in AR, much less has looked at how to author these tutorials. Beyond the technical requirements of an authoring interface, an ideal tutorial may look different depending on the end user of the tutorial. This problem is exacerbated in AR as there are many different modalities in which tutorial content can be presented. While one person may appreciate guiding animations in AR, another may prefer static text and images, and yet another may prefer video tutorials from one or multiple perspectives. With AuthAR, we present a system for building tutorials for assembly tasks that can accommodate the needs of these different types of end users. AuthAR generates video, and pictorial representations semi-automatically while the tutorial author completes the task. Furthermore, AuthAR allows tutorial authors to create and refine a tutorial in situ, integrating content authoring into the process of completing the task. This approach adds little additional overhead and reduces the need for post-processing of the tutorial. This paper presents the AuthAR system for generating mixed media assembly tutorials. Informed by prior work on content/tutorial authoring, and tutorial playback and walkthrough, we build the system with an eye toward non-obtrusive content authoring and generation of important components for tutorial playback, summarized in a set of design guidelines. We validate the system's ability to create a tutorial by stepping through the process of creating a tutorial to build a laptop stand, automatically generating an XML representation of the tutorial. Initial observations suggest the tool will be valuable, and possible ways the system could be extended and refined in future iterations. AuthAR builds on prior research in the areas of AR tutorials and content authoring, as well as principles of mixed media tutorial design. AR is often applied to assembly tasks for its ability to project instructions into the environment such that they are spatially relevant. Radkowski et al. demonstrate assembly of a 3D puzzle and provide evidence that AR supports faster assembly than traditional methods. Similarly, Henderson et al. demonstrate the use of projected guidance for engine assembly. Prior work on AR guidance design has shown that abstract representations (3D text, arrows) can be more effective for complex tasks than a more concrete representation (virtual models of the pieces and fasteners), which is sensible for simpler tasks. In that work, the authors found information-rich 2D representations to be more effective than either of the AR representations in some cases. One theory is that AR is only justified when the task is sufficiently difficult, such that the time to process the information is insignificant compared to the time to perform the task. So even for physical tasks in which instructions' spatial relevance could be increased by projecting into the environment, tutorials should provide users the ability to view the step(s) in a more familiar picture/video format when needed or preferred. The need for these mixed media tutorials is apparent, however little work has explored authoring of these tutorials for physical tasks. Outside of the realm of tutorial authoring, numerous systems have explored content creation in augmented reality to abstract low-level programming from the creator, in turn lowering the threshold for participation with AR. Many AR content authoring systems give users a collection of 3D models to place and manipulate in an environment or to overlay on a video stream, allowing users to create AR content without programming expertise generally required to build such scenes. Other AR content creation systems target specific end users for participation by domain experts in areas such as museum exhibition curation tour guidance, and assembly/maintenance. Mota et al. provides a survey of existing AR content creation tools, classifying as standalone applications vs. plug-ins to existing applications and platform-specific vs. platform-independent. Of particular interest to us are authoring tools that allow for creation of training experiences. Built on Amire's component-based framework, Zauner et al. presents a tool to enable authoring of assembly task guides in augmented reality. With this tool, the author puts visually tracked pieces together hierarchically to create an assembly workflow assistant in AR. Alternatively, the expert can collaborate remotely, rather than creating a training experience ahead of time. In this scenario, the expert can annotate the live video feed provided by the trainee's AR headset for varying levels of guidance on-the-fly. These systems require explicit enumeration of every component to be added to the created scene whereas our system generates content semi-automatically (segmenting video, recording changes to transform and detecting use of a tool) where possible. Moreover, our system only requires this manual input for augmentation and refinement of the tutorial while the bulk of authoring is done in situ. Within the domain of software tutorials, Chi et al. provides design guidelines for mixed media tutorial authoring with their MixT system for software tutorials. They list scannable steps, legible videos, visualized mouse movement and giving control to the user on which format to view as important components of a mixed media tutorial. Carter et al. echos this sentiment, with their ShowHow system for building a tutorial of videos and pictures taken from an HMD noting that mixing different media is important in lieu of relying on a single media type. Our work builds upon these concepts of mixed media tutorials but applies them to AR authoring of physical task tutorials. The rest of this subsection discusses the use of three popular media used for both software and physical task tutorials: videos, images, and interactive guidance. Prior work on video-based tutorials has applied different strategies to the challenges of video segmentation and multiple perspectives. DemoCut allows for semi-automatic video segmentation such that these demonstration videos are appropriately concise without requiring significant postprocessing. Chronicle allows for video tutorials based on the working history of a file. As the file changes, their system generates a video tutorial of how the file changed. For physical tasks, Nakae et al. propose use of multiple video perspectives (1 st person, 3 rd person, overhead) to record fabrication demonstrations and semi-automatic generation of these videos. Prior work also uses captioned and augmented images for physical task and software tutorials. Image-based tutorials can be generated automatically from demonstration as is done with TutorialPlan, a system that builds tutorials to help novices learn AutoCAD. Images can also represent groups of significant manipulations (such as multiple changes to the saturation parameter) as Grabler et al. demonstrated for GIMP tutorials. For physical tasks, prior work explored use of AR to retarget 2D technical documentation onto the object itself, providing spatially relevant augmentations. Interactive tutorials guiding users where to click, place objects, and even move their hands have become increasingly popular. For example, EverTutor automatically generates tutorials for smart phone tasks such as setting a repeating alarm or changing font size based on touch events. They found improved performance and preference toward these tutorials over text, video, and image tutorials. In the realm of physical assembly tutorials, use of visual and/or depth tracking allows for automatic generation of tutorials based on changes to location and rotation. Further, interactive tutorial authoring can include tracking of hand position, and project green and red onto end users' hands to indicate correct and incorrect positioning respectively. DuploTrack allows users to author a tutorial for creating Duplo block models using depth sensing to infer positions and rotations automatically, and projective guidance is available to end users of the tutorial. Our design of the AuthAR system was grounded by our exploration of the assembly task design space, and a study of related research and investigation into the difficulties associated with the process of generating tutorials (both for "traditional" media, as well as AR). Below we describe the design guidelines we followed when making decisions about the implementation of our system. It is important that the author and assembler be able to perform the assembly task without being burdened by the tutorial apparatus or interface. Though many AR/VR interfaces are either mediated by a handheld device or require use of freehand gestures for input, assembly tasks often require the use of both hands in parallel. For this reason, we prioritize hands-free interaction with the system such that users can always keep their hands free for assembly. Prior studies have shown that different representations (text, static pictures, video, animations, etc.) can all be valuable for people following along with a tutorial. Our system should allow authors to document their tutorial using multiple media types to best capture the necessary information. Manual content creation in AR allows for high expressivity but is time-consuming and can add complexity. Automatic content creation tools are easier to use, however have the side effect of limiting the author's creative control. Our tool should let authors move been automatic and manual creation modes to get the benefits of both models. In the most "automatic" case, an author should be able to generate a tutorial by doing little more than simply completing the assembly task as they would normally. With the ability to generate, refine and augment tutorial step representations, our tool should allow authors to create the tutorial while performing the task and make necessary tweaks directly after each step. This form of contextual and in situ editing allows authors to add callouts or make changes while they are easy to remember and reduces the need for post-processing of tutorials at a desktop computer. The design space of tutorials for assembly tasks in augmented reality is broad, with many dimensions of variability both in how the instructions are presented to an end-user and how they are authored. To inform the design of our system and explore how to best achieve our design goals, we first mapped out some areas of this design space most relevant to our work (Figure 2). Perhaps the most important dimension of variability in the AR-based assembly task tutorial design space is how the tutorial information is presented within the AR environment. Like more "traditional" tutorial delivery mediums, the AR environment is able to present static text and images, as well as show explanatory videos. A most straightforward (and still useful) application of AR technology for sharing assembly tutorials would be to display relevant information about the task as a heads-up display (HUD) in the user's headset, leaving their hand's free to perform the task. Unique to AR however is the ability to spatially associate these "traditional" elements with points or objects in the physical space. Further, an AR tutorial can present the assembly instructions "live", by displaying assembly guidance graphics or animations interleaved into the physical space. With our system we chose to use traditional text, pictures, and video presentation methods in addition to dynamic instructions, since they each have their unique benefits (D2). In order for the HoloLens to listen for the keyword "Stop Recording" during first/third person video recording, it cannot simultaneously record dictation to complement the video. For this reason, the current form of AuthAR records muted videos, but future iterations with additional microphone input capability would rectify this. With use of audio in content authoring infeasible, text serves to supplement muted video. When it comes to authoring the tutorial, the content could be constructed in situ, that is, in the same physical space as the task is being performed, or in an external secondary location such as a desktop computer. Our work explores in situ authoring to reduce the required effort (D3) and maintain context (D4). The content for the tutorial can be automatically captured as the authors moves through the assembly steps, or manually created by explicitly specifying what should be happening at each step. Automatically generating the tutorials streamlines the authoring and content creation process, but limits expressivity of the tutorial instructions. We automatically capture as much as possible, while allowing manual additions where the author thinks they would be helpful (D3). Another aspect of tutorial authoring is how the content is edited. A traditional practice is to capture the necessary content first, then go back and edit for time and clarity in post-processing. Alternately, the content can be edited concurrently with the process of content collection-which if implemented poorly could negatively impact the "flow" of the creation process. In the best case, this in having a completed tutorial ready to share as soon as the author has completed the task themselves. We focus on creating a welldesigned concurrent editing process (D1). Interacting with the tutorial system, both in the creation of the tutorial as well as when following along, can be accomplished through many means. AR applications can use touch controls, gestures, or dedicated hardware controllers. However, for assembly tasks it is desirable to have both hands available at all times for the assembly task -rather than have them occupied by control of the tutorial authoring system (D1). For that reason, we exclusively use voice and gaze controls. AuthAR is a suite of software tools that allow tutorial authors to generate AR content (Figure 1). Optitrack motion capture cameras visually track marked assembly materials and a screwdriver, adding changes to position, transform and locations of screws added with a tracked screwdriver to the tutorial automatically. The HoloLens captures first person videos and images and a mounted Android tablet simultaneously captures third person video. The HoloLens also guides authors through the process of creating the tutorial and allows for gaze-and voice-based interaction to add and update content. This includes addition of augmentations such as location-specific callout points, marking locations of untrackable screws, and specification of images as "negative examples" (steps that should be avoided) or warnings. AuthAR's system architecture consists of three components-the Microsoft HoloLens, a Samsung Tab A 10.1 Android tablet and a server running on a desktop computer all on the same network (Figure 3). As a proxy to sufficient object recognition and point-cloud generation directly from the headset, we use relatively small (~1cm) Optitrack visual markers for detection of material position and rotation. In developing AuthAR, we envisioned headsets of the future having onboard object recognition, though as a proxy to such capabilities, we implement a networked system to coordinate object positions generated by Optitrack's Motive software and the HoloLens to make the physical objects interactive. To provide this interactivity, the HoloLens generates virtual replicas of each piece and overlays these models as invisible renderings at the position and rotation of the tracked object. This invisible object takes the same shape as the physical object but has its visual rendering component disabled. It engages the HoloLens raycast and gives the user the illusion of placing virtual augmentations on physical objects. Additionally, we add a tracked handle to a screwdriver for tracking to infer screw events. The server connects to the Optitrack host via UDP and continues to update object and tool transforms. This server sends the data to the HoloLens via 64-character messages and the HoloLens' representation of the transform is updated accordingly. The Android tablet simply serves to record 3 rd person video of the tutorial. When the HoloLens starts and stops recording 1 st person demonstration, a message is passed to the server and then to the Android tablet to toggle recording. Throughout the paper, we demonstrate usage of AuthAR to build a tutorial for assembly of an Ikea laptop stand. Parts have been outfitted with Optitrack visual markers and defined as rigid bodies in Optitrack's Motive software. Simple representations of these parts are passed to the HoloLens as invisible colliders, so the physical components act as interactive objects in AR (Figure 4). Though this configuration is specific to the laptop stand, this approach is easily extensible to other assembly workflows. Users simply define combinations of visual markers as rigid bodies and build simplified models of the individual parts. In this case, these models are combinations of cubes that have been scaled along X, Y and Z axes, giving rough approximations of the parts' shapes. This initial step of predefining the shapes of these parts allows fully in situ editing allows for the remainder of the process. To avoid encumbering the assembly process with tutorial generation steps, interaction with AuthAR involves only voice, gaze, and use of materials and tools. This allows the user to always keep his/her hands free to build. By using only voice-and gaze-based interaction, we also eliminate the need for an occluding visual interface of menus and buttons, avoiding interference with the physical tutorial tasks. For flexibility, the user can add augmentations at any point while building the tutorial. However, to encourage faster onboarding, we guide the user through a two-phase process for each step: Step Recording and Step Review. When the user says "Start Recording" AuthAR begins recording changes to object transforms such that moving the physical objects maps directly to manipulating the virtual representations of those objects in the tutorial. This command also initiates video recording using the HoloLens' built-in camera and the tablet's 3 rd person perspective (Figure 5). AuthAR also records when the screwdriver's tip comes in contact with an object piece and generates a screw hole on that object (though not displayed until the Review Mode). To do this we add a tracked attachment to the handle, similar to what was done for Dodecapen and SymbiosisSketch to provide the position of the tip based on the orientation of the attachment. Given real-time streaming of position data of the screwdriver's handle and a priori knowledge of the length from the handle to the tip as 10.5 cm, we calculate the position of the screwdriver's tip as 10.5 cm forward from the handle. When the user says "Finish Recording", the HoloLens prompts the user for a step description and records dictation. When description is complete, the user enters an idle state until they are ready for review. Step Review: Manually Added Features After a tutorial author has completed a step recording, they can they can say "Review Step" to enter review mode for the step. The 1 st person video just recorded is played on a loop across from the author, automatically repositioning itself such that the author can look up at any point and see the video directly in front of him/her. This allows the author to draw upon their own experience when adding manual augmentations. Existing augmentations shift into focus by getting larger or expanding when the user is looking closer to that augmentation than any other-this eliminates the need to focus on small points (~3cm) to engage with them. When looking toward a particular object, the available commands to update the augmentation are shown in the top right of the Heads-up Display. After recording the step, the tutorial author may want to draw the tutorial user's attention to a particular point, similar to how this is done on 2D paper-based instructions (Figure 6). To do this, the author focuses the gaze-based cursor on the specific point on the tracked object where the callout point should go and says "Add Point". This adds a small virtual sphere anchored by its relative position on the object. The process of adding a captioned image (Figure 7), begins when the user looks toward a callout point and says "Add Picture". The system then starts a countdown from three in the middle of the heads-up display to indicate that a picture is about to be taken, and then displays "Hold still!" when the countdown is complete, at which point the HoloLens captures the current frame. The collected image is saved and immediately loaded over the callout point. The author can then say "Add Text", and a prompt in the middle of the heads-up display shows "Speak the image caption" and the HoloLens begins listening for recorded dictation. After a pause of 3 seconds, the HoloLens finishes recording and immediately associates the spoken text as the image caption. Authors can describe a callout point as a negative example or a warning simply by looking toward the callout point and saying "Warning". This defines that callout point as a warning or negative example and will draw tutorial users to be extra attentive when traversing the tutorial. The callout point turns red and the associated image is given a red border around it (Figure 8). Authors are also able to move the point along the surface of any tracked piece or delete it entirely. Though fastening components together is an important component of an assembly tutorial, the objects are too small to track with traditional object tracking technologies. During step recording, AuthAR records use of the tracked screwdriver and detects when it was used on a tracked material, generating a screw hole. Making use of these generated screw holes from step recording, the author can associate a virtual screw with the hole by looking toward the hole and saying "Add Screw" (Figure 9). The user cycles through possible screws by saying "Next" and "Previous"-commands which cycle through possible virtual screw representations. The author is able to hold the physical screw up to the virtual one for comparison and say "This One" to associate the screw with that hole. Authors can also manually add new fasteners in areas that cannot be tracked. To do so, the author once again says "Add Screw", pulling up the same menu, but when a screw has been selected, the ray-casted, gaze-based cursor allows the user to manually place the virtual screw where it was physically placed. This is useful for the laptop stand, for example, as there are rubber feet that need to be screwed in by hand, rather than with the tracked screwdriver. Saying "Finish Review" brings the user back to the original idle state and the user has iterated through the step building process at this point. Toward validating AuthAR, we discuss our initial observations in testing with tutorial authors, present an example application that parses and displays the generated tutorial for end users, and explain extensibility beyond the presented use case. In doing so, we consider improvements to AuthAR, and design considerations for other in situ AR content authoring tools. To gather initial observations and feedback of the system, we asked two users to generate a tutorial for an Ikea laptop stand. Though we have not formally evaluated AuthAR for usability, this initial feedback provides some insight into possible improvements to AuthAR and generalized takeaways in building such systems. The standard paper instructions for the stand consists of four steps: fastening the legs to the bottom base, fastening the top to the legs, adding screw-on feet to the bottom of the structure and adding glass to the top. We guided the users through using the system while they created an AR tutorial for assembling this piece of furniture. Both testers found the system helpful for generating tutorials, one noting that "being able to take pictures with the head for annotations was really useful." This suggests that the embodied gaze-based interaction is particularly well-suited to picture and video recording. Most of the functionality for making refinements in the tutorial is enabled by the user looking anywhere near the objects, however, adding new callout points requires accurate hovering of the cursor on the object of interest while speaking a command. One user mentioned that it was "kind awkward to point at certain points with the head". In such systems that require precise placement of virtual objects on physical components, pointing at and touching the position where the callout point should go would be a useful improvement over a gaze-only approach. Though fulfilling the hands-free requirement of a tutorial generation system (D1), AuthAR's use of dictation recognition for text entry was particularly challenging for users, in part due to the automated prompting for step descriptions and titles. One participant was surprised by the immediate prompt for a step description, and that "it was hard to formulate something articulate to say by the time it had finished recording", so future iterations will likely incorporate users explicitly starting dictation recognition for a title so they are prepared to give one. While we focus on authoring the tutorial in real time, the users wanted to view and refine previous steps. One possible enhancement to an in situ content authoring tool would be a "navigation prompt to let you jump between steps and review your work." With AuthAR, this would provide users the ability to review and refine the full tutorial at a high level, including previous steps. Figure 9. User adding virtual screws to the tutorial. The user can hold the physical screw up to the virtual one for comparison (top), and if the screw hole was not automatically generated, the user can place the screw via ray-cast from the headset (middle). Though we focus on authoring of assembly tutorials, we also implemented and tested a playback mode to validate AuthAR's ability to generate working tutorials. The tutorial author can save a tutorial after finishing a step, and AuthAR generates an XML representation of the entire tutorial. We load this XML into the playback application, also built for the HoloLens, and components of the steps built earlier are loaded and displayed for the end user to follow. The guidance techniques are simple but demonstrate success of the authoring process and portability of the generated tutorial. This application projects lines from each piece's location to where that piece needs to go (Figure 10a). When the end user of the tutorial has correctly placed the object, a notification appears in front of them (Figure 10b). The user also receives guidance of where add screws and when the playback application recognizes a screw event in that location, the screw augmentation disappears (Figure 10c). Both first person and third person video representations play on a loop across the table from the user (Figure 10d). As the user walks around the table, the videos adjust their positions such that the user can always look up and see the videos straight across from them. Use of the third person video is currently the only tutorial component that requires postprocessing after the in situ authoring process is complete. Because the XML representation of the tutorial uses file paths for videos, the author needs to manually move the video from the Android tablet to the headset's file system. Future iterations could automatically stream these videos between components. We demonstrate AuthAR with assembly of an Ikea laptop stand but note that it could be extended to any physical task where simplified virtual models of the pieces could be built or obtained. All of the pieces used for the laptop stand were scaled cubes or combinations of scaled cubes, with disabled "Renderer" components to create the illusion of adding virtual objects to physical parts, when in reality, there were invisible objects overlaid on top of the physical pieces. So simply loading in pieces and turning any visual rendering allow for extensibility to virtually any assembly task. While demonstrated with an augmented screwdriver, AuthAR could be extended to support different tools, or an integrated workspace of smart tools. We employed very simple logic, that whenever the tip of the screwdriver hovers near a piece, AuthAR adds a screw hole. Future iterations could employ more advanced logic for detecting tool usage. The need for a large truss with 10 Optitrack cameras enables very accurate tracking of visual markers but limits the AuthAR's widespread deployment in its current state. For practical use we envision the localization of objects being done with the headset only, or with cheaper external optical tracking, perhaps with black and white fiduciary markers. In this scenario, tracked pieces could be established in real time though user-assisted object recognition, rather than defining their shapes prior to running AuthAR. The in situ authoring approach offered by AuthAR allows the user to concurrently craft the tutorial while assembling the pieces. However, the gaze/voice multimodal interface does not provide users with efficient tools to fine-tune the generated tutorial. To this end, AuthAR should be complimented by a 2D interface for precise editing of individual components of the tutorial, similar to tutorial generation tools described in previous work. Trimming 1 st and 3 rd person perspective videos, cropping images and editing text are currently not well-supported by AuthAR and would be better suited to a mouse and keyboard interface. This complimentary approach would also allow users to focus on coarse-grain tutorial generation and object assembly without being burdened by these smaller edits that can easily be done after the fact. AuthAR enables tutorial authors to generate mixed media tutorials semi-automatically to guide end users through the assembly process. We automatically record expert demonstration where possible and allow for in situ editing for refinements and additions. We built AuthAR with several design guidelines in mind, validated with the authoring of a tutorial for assembling a laptop stand, and discuss the extensibility to assembly of other tasks by simply loading different virtual models into AuthAR. We see AuthAR enabling authoring of tutorials that could reach a widespread population with mixed media tutorials flexible to the preferences of each individual user."

A new algorithm to train deep neural networks. Tested on optimization functions and MNIST.

Represent the short summary for finding the corresponding scientific article (long):

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the scientific article to find a short TLDR

Question: Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies. Unfortunately, due to the large number of weights, all the examples in a mini-batch typically share the same weight perturbation, thereby limiting the variance reduction effect of large mini-batches. We introduce flipout, an efficient method for decorrelating the gradients within a mini-batch by implicitly sampling pseudo-independent weight perturbations for each example. Empirically, flipout achieves the ideal linear variance reduction for fully connected networks, convolutional networks, and RNNs. We find significant speedups in training neural networks with multiplicative Gaussian perturbations. We show that flipout is effective at regularizing LSTMs, and outperforms previous methods. Flipout also enables us to vectorize evolution strategies: in our experiments, a single GPU with flipout can handle the same throughput as at least 40 CPU cores using existing methods, equivalent to a factor-of-4 cost reduction on Amazon Web Services. Stochasticity is a key component of many modern neural net architectures and training algorithms. The most widely used regularization methods are based on randomly perturbing a network's computations BID29 BID7. Bayesian neural nets can be trained with variational inference by perturbing the weights BID4 BID0. Weight noise was found to aid exploration in reinforcement learning BID20 BID2. Evolution strategies (ES) minimizes a black-box objective by evaluating many weight perturbations in parallel, with impressive performance on robotic control tasks BID25. Some methods perturb a network's activations BID29 BID7, while others perturb its weights BID4 BID0 BID20 BID2 BID25. Stochastic weights are appealing in the context of regularization or exploration because they can be viewed as a form of posterior uncertainty about the parameters. However, compared with stochastic activations, they have a serious drawback: because a network typically has many more weights than units, it is very expensive to compute and store separate weight perturbations for every example in a mini-batch. Therefore, stochastic weight methods are typically done with a single sample per mini-batch. In contrast, activations are easy to sample independently for different training examples within a mini-batch. This allows the training algorithm to see orders of magnitude more perturbations in a given amount of time, and the variance of the stochastic gradients decays as 1/N, where N is the mini-batch size. We believe this is the main reason stochastic activations are far more prevalent than stochastic weights for neural net regularization. In other settings such as Bayesian neural nets and evolution strategies, one is forced to use weight perturbations and live with the ing inefficiency. In order to achieve the ideal 1/N variance reduction, the gradients within a mini-batch need not be independent, but merely uncorrelated. In this paper, we present flipout, an efficient method for decorrelating the gradients between different examples without biasing the gradient estimates. Flipout applies to any perturbation distribution that factorizes by weight and is symmetric around 0-including DropConnect, multiplicative Gaussian perturbations, evolution strategies, and variational Bayesian neural nets-and to many architectures, including fully connected nets, convolutional nets, and RNNs. In Section 3, we show that flipout gives unbiased stochastic gradients, and discuss its efficient vectorized implementation which incurs only a factor-of-2 computational overhead compared with shared perturbations. We then analyze the asymptotics of gradient variance with and without flipout, demonstrating strictly reduced variance. In Section 4, we measure the variance reduction effects on a variety of architectures. Empirically, flipout gives the ideal 1/N variance reduction in all architectures we have investigated, just as if the perturbations were done fully independently for each training example. We demonstrate speedups in training time in a large batch regime. We also use flipout to regularize the recurrent connections in LSTMs, and show that it outperforms methods based on dropout. Finally, we use flipout to vectorize evolution strategies BID25, allowing a single GPU to handle the same throughput as 40 CPU cores using existing approaches; this corresponds to a factor-of-4 cost reduction on Amazon Web Services. We use the term "weight perturbation" to refer to a class of methods which sample the weights of a neural network stochastically at training time. More precisely, let f (x, W) denote the output of a network with weights W on input x. The weights are sampled from a distribution q θ parameterized by θ. We aim to minimize the expected loss DISPLAYFORM0, where L is a loss function, and D denotes the data distribution. The distribution q θ can often be described in terms of perturbations: W = W + ∆W, where W are the mean weights (typically represented explicitly as part of θ) and ∆W is a stochastic perturbation. We now give some specific examples of weight perturbations. Gaussian perturbations. If the entries ∆W ij are sampled independently from Gaussian distributions with variance σ 2 ij, this corresponds to the distribution W ij ∼ N (W ij, σ 2 ij). Using the reparameterization trick BID9, this can be rewritten as W ij = W ij + σ ij ij, where ij ∼ N; this representation allows the gradients to be computed using backprop. A variant of this is multiplicative Gaussian perturbation, where the perturbations are scaled according to the weights: DISPLAYFORM1, where again ij ∼ N. Multiplicative perturbations can be more effective than additive ones because the information content of the weights is the same regardless of their scale. DropConnect. DropConnect BID30 ) is a regularization method inspired by dropout BID29 which randomly zeros out a random subset of the weights. In the case of a 50% drop rate, this can be thought of as a weight perturbation where W = W/2 and each entry ∆W ij is sampled uniformly from ±W ij.Variational Bayesian neural nets. Rather than fitting a point estimate of a neural net's weights, one can adopt the Bayesian approach of putting a prior distribution p(W) over the weights and approximating the posterior distribution p(W |D) ∝ p(W)p(D|W), where D denotes the observed data. BID4 observed that one could fit an approximation q θ (W) ≈ p(W |D) using variational inference; in particular, one could maximize the evidence lower bound (ELBO) with respect to θ: DISPLAYFORM2 The negation of the second term can be viewed as the description length of the data, and the negation of the first term can be viewed as the description length of the weights BID6. BID4 observed that if q is chosen to be a factorial Gaussian, sampling from θ can be thought of as Gaussian weight perturbation where the variance is adapted to maximize F. BID0 later combined this insight with the reparameterization trick BID9 to derive unbiased stochastic estimates of the gradient of F.Evolution strategies. ES BID22 ) is a family of black box optimization algorithms which use weight perturbations to search for model parameters. ES was recently proposed as an alternative reinforcement learning algorithm BID26 BID25. In each iteration, ES generates a collection of weight perturbations as candidates and evaluates each according to a fitness function F. The gradient of the parameters can be estimated from the fitness function evaluations. ES is highly parallelizable, because perturbations can be generated and evaluated independently by different workers. Suppose M is the number of workers, W is the model parameter, σ is the standard deviation of the perturbations, α is the learning rate, F is the objective function, and ∆W m is the Gaussian noise generated at worker m. The ES algorithm tries to maximize E∆W F W + σ∆W. The gradient of the objective function and the update rule can be given as: DISPLAYFORM3 In some cases, it's possible to reformulate weight perturbations as activation perturbations, thereby allowing them to be efficiently computed fully independently for different examples in a mini-batch. In particular, BID10 showed that for fully connected networks with no weight sharing, unbiased stochastic gradients could be computed without explicit weight perturbations using the local reparameterization trick (LRT). For example, suppose X is the input mini-batch, W is the weight matrix and B = XW is the matrix of activations. The LRT samples the activations B rather than the weights W. In the case of a Gaussian posterior, the LRT is given by: DISPLAYFORM0 where b m,j denotes the perturbed activations. While the exact LRT applies only to fully connected networks with no weight sharing, BID10 also introduced variational dropout, a regularization method inspired by the LRT which performs well empirically even for architectures the LRT does not apply to. Control variates are another general class of strategies for variance reduction, both for black-box optimization BID31 BID21 BID19 and for gradient-based optimization BID24 BID18 BID15. Control variates are complementary to flipout, so one could potentially combine these techniques to achieve a larger variance reduction. We also note that the fastfood transform BID13 ) is based on similar mathematical techniques. However, whereas fastfood is used to approximately multiply by a large Gaussian matrix, flipout preserves the random matrix's distribution and instead decorrelates the gradients between different samples. As described above, weight perturbation algorithms suffer from high variance of the gradient estimates because all training examples in a mini-batch share the same perturbation. More precisely, sharing the perturbation induces correlations between the gradients, implying that the variance can't be eliminated by averaging. In this section, we introduce flipout, an efficient way to perturb the weights quasi-independently within a mini-batch. We make two assumptions about the weight distribution q θ: the perturbations of different weights are independent; and the perturbation distribution is symmetric around zero. These are nontrivial constraints, but they encompass important use cases: independent Gaussian perturbations (e.g. as used in variational BNNs and ES) and DropConnect with drop probability 0.5. We observe that, under these assumptions, the perturbation distribution is invariant to elementwise multiplication by a random sign matrix (i.e. a matrix whose entries are ±1). In the following, we denote elementwise multiplication by •. Observation 1. Let q θ be a perturbation distribution that satisfies the above assumptions, and let ∆W ∼ q θ. Let E be a random sign matrix that is independent of ∆W. Then ∆W = ∆W • E is identically distributed to ∆W. Furthermore, the loss gradients computed using ∆W are identically distributed to those computed using ∆W.Flipout exploits this fact by using a base perturbation ∆W shared by all examples in the mini-batch, and multiplies it by a different rank-one sign matrix for each example: DISPLAYFORM0 where the subscript denotes the index within the mini-batch, and r n and s n are random vectors whose entries are sampled uniformly from ±1. According to Observation 1, the marginal distribution over gradients computed for individual training examples will be identical to the distribution computed using shared weight perturbations. Consequently, flipout yields an unbiased estimator for the loss gradients. However, by decorrelating the gradients between different training examples, we can achieve much lower variance updates when averaging over a mini-batch. Vectorization. The advantage of flipout over explicit perturbations is that computations on a minibatch can be written in terms of matrix multiplications. This enables efficient implementations on GPUs and modern accelerators such as the Tensor Processing Unit (TPU) . Let x denote the activations in one layer of a neural net. The next layer's activations are given by: DISPLAYFORM1 where φ denotes the activation function. To vectorize these computations, we define matrices R and S whose rows correspond to the random sign vectors r n and s n for all examples in the mini-batch. The above equation is vectorized as: DISPLAYFORM2 This defines the forward pass. Because R and S are sampled independently of W and ∆W, we can backpropagate through Eqn. 4 to obtain derivatives with respect to W, ∆W, and X.Computational cost. In general, the most expensive operation in the forward pass is matrix multiplication. Flipout's forward pass requires two matrix multiplications instead of one, and therefore should be roughly twice as expensive as a forward pass with a single shared perturbation when the multiplications are done in sequence. 1 However, note that the two matrix multiplications are independent and can be done in parallel; this incurs the same overhead as the local reparameterization trick BID10.A general rule of thumb for neural nets is that the backward pass requires roughly twice as many FLOPs as the forward pass. This suggests that each update using flipout ought to be about twice as expensive as an update with a single shared perturbation (if the matrix multiplications are done sequentially); this is consistent with our experience. Evolution strategies. ES is a highly parallelizable algorithm; however, most ES systems are engineered to run on multi-core CPU machines and are not able to take full advantage of GPU parallelism. Flipout enables ES to run more efficiently on a GPU because it allows each worker to evaluate a batch of quasi-independent perturbations rather than only a single perturbation. To apply flipout to ES, we can simply replicate the starting state by the number of flipout perturbations N, at each worker. Instead of Eqn. 1, the update rule using M workers becomes: DISPLAYFORM3 where m indexes workers, n indexes the examples in a worker's batch, and F mn is the reward evaluated with the n th perturbation at worker m. Hence, each worker is able to evaluate multiple perturbations as a batch, allowing for parallelism on a GPU architecture. In this section, we analyze the variance of stochastic gradients with and without flipout. We show that flipout is guaranteed to reduce the variance of the gradient estimates compared to using naïve shared perturbations. DISPLAYFORM0 ) under the perturbation ∆W for a single training example x. (Note that G x is a random variable which depends on both x and ∆W . We analyze a single entry of the gradient so that we can work with scalar-valued variances.) We denote the gradient averaged over a mini-batch as the random variable DISPLAYFORM1 denotes a mini-batch of size N, and ∆W n denotes the perturbation for the n th example. (The randomness comes from both the choice of B and the random perturbations.) For simplicity, we assume that the x n are sampled i.i.d. from the data distribution. Using the Law of Total Variance, we decompose Var(G B) into a data term (the variance of the exact mini-batch gradients) and an estimation term (the estimation variance for a fixed mini-batch): DISPLAYFORM2 Notice that the data term decays with N while the estimation term may not, due to its dependence on the shared perturbation. But we can break the estimation term into two parts for which we can analyze the dependence on N. To do this, we reformulate the standard shared perturbation scheme as follows: ∆W is generated by first sampling ∆W and then multiplying it by a random sign matrix rs as in Eqn. 3 -exactly like flipout, except that the sign matrix is shared by the whole minibatch. According to Observation 1, this yields an identical distribution for ∆W to the standard shared perturbation scheme. Based on this, we obtain the following decomposition: Theorem 2 (Variance Decomposition Theorem). Define α, β, and γ to be DISPLAYFORM3 DISPLAYFORM4 Under the assumptions of Observation 1, the variance of the gradients under shared perturbations and flipout perturbations can be written in terms of α, β, and γ as follows:Fully independent perturbations: DISPLAYFORM5 Flipout: DISPLAYFORM6 Proof. Details of the proof are provided in Appendix A.We can interpret α, β, and γ as follows. First, α combines the data term from Eqn. 6 with the expected estimation variance for individual data points. This corresponds to the variance of the gradients on individual training examples, so fully independent perturbations yield a total variance of α/N. The other terms, β and γ, reflect the covariance between the estimation errors on different training examples as a of the shared perturbations. The term β reflects the covariance that from sampling r and s, so it is eliminated by flipout, which samples these vectors independently. Finally, γ reflects the covariance that from sampling ∆W, which flipout does not eliminate. Empirically, for all the neural networks we investigated, we found that α β γ. This implies the following behavior for Var(G B) as a function of N: for small N, the data term α/N dominates, giving a 1/N variance reduction; with shared perturbations, once N is large enough that α/N < β, the variance Var(G B) levels off to β. However, flipout continues to enjoy a 1/N variance reduction in this regime. In principle, flipout's variance should level off at the point where α/N < γ, but in all of our experiments, γ was small enough that this never occurred: flipout's variance was approximately α/N throughout the entire range of N values we explored, just as if the perturbations were sampled fully independently for every training example. We first verified empirically the variance reduction effect of flipout predicted by Theorem 2; we measured the variance of the gradients under different perturbations for a wide variety of neural network architectures and batch sizes. In Section 4.2, we show that flipout applied to Gaussian perturbations and DropConnect is effective at regularizing LSTM networks. In Section 4.3, we demonstrate that flipout converges faster than shared perturbations when training with large minibatches. Finally, in Section 4.4 we present experiments combining Evolution Strategies with flipout in both supervised learning and reinforcement learning tasks. In our experiments, we consider the four architectures shown in TAB1 (details in Appendix B). Since the main effect of flipout is intended to be variance reduction of the gradients, we first estimated the gradient variances of several architectures with mini-batch sizes ranging from 1 to 8196 FIG0 ). We experimented with three perturbation methods: a single shared perturbation per minibatch, the local reparameterization trick (LRT) of BID10, and flipout. For each of the FC, ConVGG, and LSTM architectures, we froze a partially trained network to use for all variance estimates, and we used multiplicative Gaussian perturbations with σ 2 = 1. We computed Monte Carlo estimates of the gradient variance, including both the data and estimation terms in Eqn. 6. Confidence intervals are based on 50 independent runs of the estimator. Details are given in Appendix C.The analysis in Section 3.2 makes strong predictions about the shapes of the curves in FIG0. By Theorem 2, the variance curves for flipout and shared perturbations each have the form a + b/N, where N is the mini-batch size. On a log-log plot, this functional form appears as a linear regime with slope -1, a constant regime, and a smooth phase transition in between. Also, because the distribution of individual gradients is identical with and without flipout, the curves must agree for N = 1. Our plots are consistent with both of these predictions. We observe that for shared perturbations, the phase transition consistently occurs for mini-batch sizes somewhere between 100 and 1000. In contrast, flipout gives the ideal linear variance reduction throughout the range of mini-batch sizes we investigated, i.e., its behavior is indistinguishable from fully independent perturbations. As analyzed by BID10, the LRT gradients are fully independent within a mini-batch, and are therefore guaranteed to achieve the ideal 1/N variance reduction. Furthermore, they reduce the variance below that of explicit weight perturbations, so we would expect them to achieve smaller variance than flipout, as shown in FIG0. However, flipout is applicable to a wider variety of architectures, including convolutional nets and RNNs. We evaluated the regularization effect of flipout on the character-level and word-level language modeling tasks with the Penn Treebank corpus (PTB) BID16. We compared flipout to several other methods for regularizing RNNs: naïve dropout BID32, variational dropout BID3, recurrent dropout BID27, zoneout BID12, and DropConnect BID17. BID32 apply dropout only to the feed-forward connections of an RNN (to the input, output, and connections between layers). The other methods regularize the recurrent connections as well: BID27 apply dropout to the cell update vector, with masks sampled either per step or per sequence; BID3 apply dropout to the forward and recurrent connections, with all dropout masks sampled per sequence. BID17 use DropConnect to regularize the hidden-to-hidden weight matrices, with a single DropConnect mask shared between examples in a mini-batch. We denote their model WD (for weight-dropped LSTM).Character-Level. For our character-level experiments, we used a single-layer LSTM with 1000 hidden units. We trained each model on non-overlapping sequences of 100 characters in batches of size 32, using the AMSGrad variant of Adam with learning rate 0.002. We perform early stopping based on validation performance. Here, we applied flipout to the hidden-tohidden weight matrix. More hyperparameter details are given in Appendix D. The , measured in bits-per-character (BPC) for the validation and test sequences of PTB, are shown in Table 2. In the table, shared perturbations and flipout (with Gaussian noise sampling) are denoted by Mult. Gauss and Mult. Gauss + Flipout, respectively. We also compare to RBN (recurrent batchnorm) and H-LSTM+LN (HyperLSTM + LayerNorm) BID5. Mult. Gauss + Flipout outperforms the other methods, and achieves the best reported for this architecture. Word-Level. For our word-level experiments, we used a 2-layer LSTM with 650 hidden units per layer and 650-dimensional word embeddings. We trained on sequences of length 35 in batches of size 40, for 100 epochs. We used SGD with initial learning rate 30, and decayed the learning rate by a factor of 4 based on the nonmonotonic criterion introduced by BID17. We used flipout to implement DropConnect, as described in Section 2.1, and call this WD+Flipout. We applied WD+Flipout to the hidden-to-hidden weight matrices for recurrent regularization, and used the same hyperparameters as BID17. We used embedding dropout (setting rows of the embedding matrix to 0) with probability 0.1 for all regularized models except Gal, where we used Table 3: Perplexity on the PTB word-level validation and test sets. All are from our own experiments. probability 0.2 as specified in their paper. More hyperparameter details are given in Appendix D. We show in Table 3 that WD+Flipout outperforms the other methods with respect to both validation and test perplexity. In Appendix E.4, we show that WD+Flipout yields significant variance reduction for large mini-batches, and that when training with batches of size 8192, it converges faster than WD. Theorem 2 and FIG0 suggest that the variance reduction effect of flipout is more pronounced in the large mini-batch regime. In this section, we train a Bayesian neural network with mini-batches of size 8192 and show that flipout speeds up training in terms of the number of iterations. We trained the FC and ConvLe networks from Section 4.1 using Bayes by Backprop BID0. Since our primary focus is optimization, we focus on the training loss, shown in FIG2 for FC, we compare flipout with shared perturbations and the LRT; for ConvLe, we compare only to shared perturbations since the LRT does not give an unbiased gradient estimator. We found that flipout converged in about 3 times fewer iterations than shared perturbations for both models, while achieving comparable performance to the LRT for the FC model. Because flipout is roughly twice as expensive as shared perturbations (see Section 3.1), this corresponds to a 1.5x speedup overall. Curves for the training and test error are given in Appendix E.2. ES typically runs on multiple CPU cores. The challenge in making ES GPU-friendly is that each sample requires computing a separate weight perturbation, so traditionally each worker can only generate one sample at a time. In Section 3.1, we showed that ES with flipout allows each worker to evaluate a batch of perturbations, which can be done efficiently on a GPU. However, flipout induces correlations between the samples, so we investigated whether these correlations cause a slowdown in training relative to fully independent perturbations (which we term "IdealES"). In this section, we show empirically that flipout ES is just as sample-efficient as IdealES, and consequently one can obtain significantly higher throughput per unit cost using flipout ES on a GPU.The ES gradient defined in Eqn. 1 has high variance, so a large number of samples are generally needed before applying an update. We found that 5,000 samples are needed to achieve stable performance in the supervised learning tasks. Standard ES runs the forward pass 5,000 times with independent weight perturbations, which sees little benefit to using a GPU over a CPU. FlipES allows the same number of samples to be evaluated using a much smaller number of explicit perturbations. Throughout the experiments, we ran flipout with mini-batches of size 40 (i.e. N = 40 in Eqn. 5).We compared IdealES and FlipES with a fully connected network (FC) on the MNIST dataset. FIG2 shows that we incur no loss in performance when using pseudo-independent noise. Next, we compared FlipES and cpuES (using 40 CPU cores) in terms of the per-update time with respect to the model size. The (in Appendix E.3) shows that FlipES scales better because it runs on the GPU. Finally, we compared FlipES and the backpropagation algorithm on both FC and ConvLe. FIG2 and FIG2 show that FlipES achieves data efficiency comparable with the backpropagation algorithm. IdealES has a much higher computational cost than backpropagation, due to the large number of forward passes. FlipES narrows the computational gap between them. Although ES is more expensive than backpropagation, it can be applied to models which are not fully differentiable, such as models with a discrete loss (e.g., accuracy or BLEU score) or with stochastic units. We have introduced flipout, an efficient method for decorrelating the weight gradients between different examples in a mini-batch. We showed that flipout is guaranteed to reduce the variance compared with shared perturbations. Empirically, we demonstrated significant variance reduction in the large batch setting for a variety of network architectures, as well as significant speedups in training time. We showed that flipout outperforms dropout-based methods for regularizing LSTMs. Flipout also makes it practical to apply GPUs to evolution strategies, ing in substantially increased throughput for a given computational cost. We believe flipout will make weight perturbations practical in the large batch setting favored by modern accelerators such as Tensor Processing Units . DISPLAYFORM0 In this section, we provide the proof of Theorem 2 (Variance Decomposition Theorem).Proof. We use the notations from Section 3.2. Let x, x denote two training examples from the mini-batch B, and ∆W, ∆W denote the weight perturbations they received. We begin with the decomposition into data and estimation terms (Eqn. 6), which we repeat here for convenience: DISPLAYFORM1 The data term from Eqn. 13 can be simplified: DISPLAYFORM2 We break the estimation term from Eqn. 13 into variance and covariance terms: DISPLAYFORM3 We now separately analyze the cases of fully independent perturbations, shared perturbations, and flipout. Fully independent perturbations. If the perturbations are fully independent, the second term in Eqn. 15 disappears. Hence, combining Eqns. 13, 14, and 15, we are left with DISPLAYFORM4 which is just α/N. Recall that we reformulate the shared perturbations in terms of first sampling ∆W, and then letting ∆W = ∆W • rs, where r and s are random sign vectors shared by the whole batch. Using the Law of Total Variance, we break the second term in Eqn. 15 into a part that comes from sampling ∆W and a part that comes from sampling r and s. DISPLAYFORM0 Since the perturbations are shared, ∆W = ∆W, so this can be simplified slightly to: DISPLAYFORM1 Plugging these two terms into the second term of Eqn. 15 yields Here, we provide details of the network configurations used for our experiments (Section 4).The FC network is a 3-layer fully-connected network with 512-512-10 hidden units. ConvLe is a LeNet-like network BID14 where the first two layers are convolutional with 32 and 64 filters of size, and use ReLU non-linearities. A 2 × 2 max pooling layer follows after each convolutional layer. Dimensionality reduction only takes place in the pooling layer; the stride for pooling is two and padding is used in the convolutional layers to keep the dimension. Two fully-connected layers with 1024 and 10 hidden units are used to produce the classification . ConVGG is based on the VGG16 network BID28. We modified the last fully connected layer to have 10 output dimensions for our experiments on CIFAR-10. We didn't use batch normalization for the variance reduction experiment since it introduces extra stochasticity. The architectures used for the LSTM experiments are described in Section 4.2. The hyperparameters used for the language modelling experiments are provided in Appendix D. Given a network architecture, we compute the empirical stochastic gradient update variance as follows. We start with a moderately pre-trained model, such as a network with 85% training accuracy on MNIST. Without updating the parameters, we obtain the gradients of all the weights by performing a feed-forward pass, that includes sampling ∆W, R, and S, followed by backpropagation. The gradient variance of each weight is computed by repeating this procedure 200 times in the experiments. Let Var lj denote the estimate of the gradient variance of weight j in layer l. We compute the gradient variance as follows: DISPLAYFORM0 where g i lj is the gradient received by weight j in layer l. We estimate the variance of the gradients in layer l by averaging the variances of the weights in that layer,Ṽ = 1 |J| j Var lj. In order to compute a confidence interval on the gradient variance estimate, we repeat the above procedure 50 times, yielding a sequence of average variance estimates, V 1,..., V 50. FIG0, we compute the 90% confidence intervals of the variance estimates with a t-test. For ConVGG, multiple GPUs were needed to run the variance reduction experiment with large mini-batch sizes (such as 4096 and 8192). In such cases, it is computationally efficient to generate independent weight perturbations on different GPUs. However, since our aim was to understand the effects of variance reduction independent of implementation, we shared the base perturbation among all GPUs to produce the plot shown in FIG0. We show in Appendix E that flipout yields lower variance even when we sample independent perturbations on different GPUs. For the LSTM variance reduction experiments, we used the two-layer LSTM described in Section 4.2, trained for 3 epochs on the word-level Penn Treebank dataset. FIG0, we split large mini-batches (size 128 and higher) into sub-batches of size 64; we sampled one base perturbation ∆W that was shared among all sub-batches, and we sampled independent R and S matrices for each sub-batch. Long Short-Term Memory networks (LSTMs) are defined by the following equations: DISPLAYFORM0 where i t, f t, and o t are the input, forget, and output gates, respectively, g t is the candidate update, and • denotes elementwise multiplication. Naïve application of dropout on the hidden state of an LSTM is not effective, because it leads to significant memory loss over long sequences. Several approaches have been proposed to regularize the recurrent connections, based on applying dropout to specific terms in the LSTM equations. BID27 propose to drop the cell update vector, with a dropout mask d t sampled either per-step or per-sequence: DISPLAYFORM1 Gal & Ghahramani FORMULA3 BID12 propose to zone out units rather than dropping them; the hidden state and cell values are either stochastically updated or maintain their previous value: DISPLAYFORM2, with zoneout masks d For the word-level models (Table 3), we used gradient clipping threshold 0.25 and the following hyperparameters:• , we used variational dropout with the parameters given in their paper: 0.35 dropout probability on inputs and outputs, 0.2 hidden state dropout, and 0.2 embedding dropout.• For BID27, we used 0.1 embedding dropout, 0.5 dropout on inputs and outputs, and 0.3 dropout on cell updates, with per-step mask sampling.• For BID12, we used 0.1 embedding dropout, 0.5 dropout on inputs and outputs, and cell and hidden state zoneout probabilities of 0.25 and 0.025, respectively. • For WD BID17, we used the parameters given in their paper: 0.1 embedding dropout, 0.4 dropout probability on inputs and outputs, and 0.3 dropout probability on the output between layers (the same masks are used for each step of a sequence). We use 0.5 probability for DropConnect applied to the hidden-to-hidden weight matrices.• For WD+Flipout, we used the same parameters as BID17, given above, but we regularized the hidden-to-hidden weight matrices with the variant of flipout described in Section 2.1, which implements DropConnect with probability 0.5.For the character-level models (Table 2), we used orthogonal initialization for the LSTM weight matrices, gradient clipping threshold 1, and did not use input or output dropout. The input characters were represented as one-hot vectors. We used the following hyperparameters for each model:• For recurrent dropout BID27, we used 0.25 dropout probability on the cell state, and per-step mask sampling.• For Zoneout BID12, we used 0.5 and 0.05 for the cell and hidden state zoneout probabilities, respectively. • For the variational LSTM BID3, we used 0.25 hidden state dropout.• For the flipout and shared perturbation LSTMs, we sampled Gaussian noise with σ = 1 for the hidden-to-hidden weight matrix. As discussed in Appendix B, training on multiple GPUs naturally induces independent noise for each sub-batch. FIG5 shows that flipout still achieves lower variance than shared perturbations in such cases. When estimating the variance with mini-batch size 8192, running on four GPUs naturally induces four independent noise samples, for each sub-batch of size 2048; this yields lower variance than using a single noise sample. Similarly, for mini-batch size 4096, two independent noise samples are generated on separate GPUs. E.2 LARGE BATCH TRAINING WITH FLIPOUT FIG6 shows the training and test error for the large mini-batch experiments described in Section 4.3. For both FC and ConvLe networks, we used the Adam optimizer with learning rate 0.003. We downscaled the KL term by a factor of 10 to achieve higher accuracy. While FIG2 shows that flipout converges faster than shared perturbations, FIG6 shows that flipout has the same generalization ability as shared perturbations (the faster convergence doesn't in overfitting). The variance reduction offered by flipout allows us to use DropConnect BID30 efficiently in a large mini-batch setting. Here, we use flipout to implement DropConnect as described in Section 2.1, and use it to regularize an LSTM word-level language model. We used the LSTM architecture proposed by BID17, which has 400-dimensional word embedddings and three layers with hidden dimension 1150. Following BID17, we tied the weights of the embedding layer and the decoder layer. BID17 use DropConnect to regularize the hidden-to-hidden weight matrices, with a single mask shared for all examples in a batch. We used flipout to achieve a different DropConnect mask per example. We applied WD+Flipout to both the hidden-to-hidden (h2h) and input-to-hidden (i2h) weight matrices, and compared to the model from BID17, which we call WD (for weight-dropped LSTM), with DropConnect applied to both h2h and i2h. Both models use embedding dropout 0.1, output dropout 0.4, and have DropConnect probability 0.5 for the i2h and h2h weights. Both models were trained using Adam with learning rate 0.001. FIG8 compares the variance of the gradients of the first-layer hidden-to-hidden weights between WD and WD+Flipout, and shows that flipout achieves significant variance reduction for mini-batch sizes larger than 256. FIG9 shows the training curves of both models with batch size 8192. We see that WD+Flipout converges faster than WD, and achieves a lower training perplexity, showcasing the optimization benefits of flipout in large mini-batch settings. Training curves for WD and WD+Flipout, with batch size 8192.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce flipout, an efficient method for decorrelating the gradients computed by stochastic neural net weights within a mini-batch by implicitly sampling pseudo-independent weight perturbations for each example.

Represent the TLDR for finding the corresponding scientific article:

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.