Question: Represent the article for finding a single-sentence TLDR

Question: The success of reinforcement learning in the real world has been limited to instrumented laboratory scenarios, often requiring arduous human supervision to enable continuous learning. In this work, we discuss the required elements of a robotic system that can continually and autonomously improve with data collected in the real world, and propose a particular instantiation of such a system. Subsequently, we investigate a number of challenges of learning without instrumentation -- including the lack of episodic resets, state estimation, and hand-engineered rewards -- and propose simple, scalable solutions to these challenges. We demonstrate the efficacy of our proposed system on dexterous robotic manipulation tasks in simulation and the real world, and also provide an insightful analysis and ablation study of the challenges associated with this learning paradigm. Reinforcement learning (RL) can in principle enable real-world autonomous systems, such as robots, to autonomously acquire a large repertoire of skills. Perhaps more importantly, reinforcement learning can enable such systems to continuously improve the proficiency of their skills from experience. However, realizing this promise in reality has proven challenging: even with reinforcement learning methods that can acquire complex behaviors from high-dimensional low-level observations, such as images, the typical assumptions of the reinforcement learning problem setting do not fit perfectly into the constraints of the real world. For this reason, most successful robotic learning experiments have been demonstrated with varying levels of instrumentation, in order to make it practical to define reward functions (e.g. by using auxiliary sensors (a; ;) ), and in order to make it practical to reset the environment between trials (e.g. using manually engineered contraptions). In order to really make it practical for autonomous learning systems to improve continuously through real-world operation, we must lift these constraints and design learning systems whose assumptions match the constraints of the real world, and allow for uninterrupted continuous learning with large amounts of real world experience. What exactly is holding back our reinforcement learning algorithms from being deployed for learning robotic tasks (for instance manipulation) directly in the real world? We hypothesize that our current reinforcement learning algorithms make a number of unrealistic assumptions that make real world deployment challenging -access to low-dimensional Markovian state, known reward functions, and availability of episodic resets. In practice, this means that significant human engineering is required to materialize these assumptions in order to conduct real-world reinforcement learning, which limits the ability of learning-enabled robots to collect large amounts of experience automatically in a variety of naturally occuring environments. Even if we can engineer a complex solution for instrumentation in one environment, the same may need to be done for every environment being learned in. When using deep function approximators, actually collecting large amounts of real world experience is typically crucial for effective generalization. The inability to collect large amounts of real world data autonomously significantly limits the ability of these robots to learn robust, generalizable behaviors. In this work, we propose that overcoming these challenges requires designing robotic systems that possess three fundamental capabilities: they are able to learn from their own raw sensory inputs, they are able to assign rewards to their own behaviors with minimal human intervention, they are able to learn continuously in non-episodic settings without requiring human operators to manually reset the environment. We believe that a system with these capabilities will bring us significantly closer to the goal of continuously improv-ing robotic agents that leverage large amounts of their own real world experience, without requiring significant human instrumentation and engineering effort. Having laid out these requirements, we propose a practical instantiation of such a learning system, which afford the above capabilities. While prior works have studied each of these issues in isolation, combining solutions to these issues is non-trivial and in a particularly challenging learning problem. We provide a detailed empirical analysis of these issues, both in simulation and on a real-world robotic platform, and propose a number of simple but effective solutions that can make it possible to produce a complete robotic learning system that can learn autonomously, handle raw sensory inputs, learn reward functions from easily available supervision, and learn without manually designed reset mechanisms. We show that this system is well suited for learning dexterous robotic manipulation tasks in the real world, and substantially outperforms ablations and prior work. While the individual components that we combine to design our robotic learning system are based heavily on prior work, both the combination of these components and their specific instantiations are novel. Indeed, we show that without the particular design decisions motivated by our experiments, naïve designs that follow prior work generally fail to satisfy one of the three requirements that we lay out. Let us start by considering the standard reinforcement learning paradigm, where we operate in a Markov decision process with state space S, action space A, unknown transition dynamics T, unknown reward function R and an episodic initial state distribution ρ. In RL, the goal is to learn a policy to maximize the expected sum of rewards via environment interactions. There is no special object instrumentation -only an RGB camera and proprioceptive sensors. Although this formalism is simple and concise, it does not capture all of the complexities of real-world robotic learning problems. If a robotic system is to learn continuously and autonomously in the real world, we must ensure that it can learn under all assumptions that are imposed by the real world. The real world does not have instrumentation available to easily provide low dimensional state estimates, rewards or episodic resets. To move from the idealized MDP formulation to the real world, we require a system that has the following properties. Firstly, all of the information necessary for learning must be obtained from the robot's own sensors. This includes all information about the state and necessitates that the policy must be learned from high-dimensional and low-level sensory observations, such as camera images. Secondly, the robot must also obtain the reward signal itself from its own sensor readings. This is exceptionally difficult for all but the simplest tasks, since reward functions that depend on affecting change in the world to specific objects require perceiving those objects explicitly. Thirdly, we must be able to learn in a non-episodic manner, without access to episodic resets. A setup with explicit resets is increasingly impractical due to the requirement for significant human engineering or intervention during learning. While some of the components discussed above can be tackled in isolation by current algorithms, there are unique challenges inherent to assembling these components into a complete learning system for real world robotics, as well as certain challenges associated with individual components. In the subsequent discussion, we outline the elements that are required to build a robotic system that can learn in the real world with minimal instrumentation. These elements present interesting challenges in learning when combined together, which we analyze in Section 3 and address in Section 4. To enable learning without complex state estimation systems or instrumenting every environment the robot operates in, we require our robotic systems to be able to learn from their own raw sensory observations. Typically, these sensory observations are raw camera images from a camera mounted on the robot and proprioceptive sensory inputs such as the joint angles. These observations do not Figure 2: Schematic comparison of current learning systems versus our proposed instrumentation-free system -R3L. While traditional robotic applications of RL in the real world require explicit supervision in the form of resets, rewards and state estimation, R3L gets rid of these requirements and allows us to learn without any explicit system instrumentation, simply by leveraging interaction with the environment. directly provide the poses of the objects in the scene, which is the typical assumption in simulated robotic environments -any such information must be extracted by the learning system. While in principle many RL frameworks can support learning from raw sensory inputs (; ;), it is important to consider the practicalities of this approach. For instance, we can instantiate vision-based RL with policy gradient algorithms such as TRPO and PPO , but these have high sample complexities which make them unsuited for real world robotic learning (a), which is further exacerbated when learning from visual inputs. In our work, we consider adopting the general framework of off-policy actor-critic reinforcement learning, using a version of the soft actor critic (SAC) algorithm described by (b). This algorithm effectively uses off-policy data, and has been shown to learn some tasks directly from visual inputs. However, while SAC is able to learn directly from raw visual input, most instantiations have required instrumentation for providing rewards and episodic resets. As we show in Section 3, when these assumptions are lifted it leads to a number of non-trivial learning challenges which require new techniques to address. Vision-based RL algorithms, such as SAC, rely on a proper reward function being provided to the system, which is typically hand-defined by a user. While this can be simple to provide in simulation, it is significantly harder to implement in uninstrumented real world environments. In the real world, the robot must obtain reward signal itself from its own sensor readings, which can be extremely challenging. A few unappealing options have been suggested to tackle this: engineer complete computer vision systems to detect objects and extract reward signals (;, engineer reward functions that use various task-specific heuristics to obtain rewards from pixels , or instrument every environment . All are highly manual, tedious processes, and a better solution is needed to scale real world robotic learning gracefully. To devise a system that requires minimal human engineering and supervision for providing rewards, we must use algorithms that are able to assign themselves rewards throughout learning with minimal reward engineering. One candidate is for a user to specify intended behavior beforehand through example images of desired goals. The algorithm can then assign itself reward based on a notion of how well it is accomplishing the specified goals during learning, with no additional human supervision. This scheme scales well since it requires minimal human engineering, and goal images are easy to provide upfront. So how exactly might we design such a reward provision system? To do this, we use a data-driven reward specification framework called variational inverse control with events (VICE), introduced by. VICE learns rewards in a task-agnostic way: we provide the algorithm with success examples in the form of images where the task is accomplished, and learn a discriminator that is capable of distinguishing successes from failures. This discriminator can then be used to provide a learning signal to nudge the reinforcement learning agent towards success. This algorithm has been previously considered in the context of learning some tasks from raw sensory observations in the real world by , but we show that it presents unique challenges when used in conjunction with learning without episodic resets. Details and specifics of the algorithms being considered are described in Appendix A and also in . While the components described in Section 2.1 and 2.2 are essential to building continuously learning RL systems in the real world, they have often been implemented with the assumption of episodic learning. Indeed, previous applications of or Haarnoja et al. (2018b) were implemented with explicitly designed reset mechanisms or human operators performing resets between trials. However, natural open-world settings do not provide any such reset mechanism, and in order to enable scalable and autonomous real-world learning, we need systems that do not require an episodic formulation of the learning problem. In principle, algorithms such as SAC do not actually require episodic learning; however, in practice, most instantiations of these algorithms have used explicit resets, even in simulation, and removing resets has ed in failure to solve challenging tasks. While RL algorithms can handle nonepisodic settings without any modifications in principle, they struggle when applied in practice to challenging problems. In our experiments in Section 3, we see that simply applying actor-critic methods to the reset free setting doesn't learn intended behaviors and requires novel insights when combined with visual observations and classifier based rewards. These three components -vision-based RL with actor-critic algorithms, vision-based goal classifier for rewards, and reset-free learning -are the fundamental pieces that we need to build a real world robotic learning system. However, when we actually combine the individual components in Sections 3 and 6, we find that learning effective policies is quite challenging. We provide insight into these challenges in Section 3 and, based on these insights, we propose a number of simple but important changes in Section 4 that enable R3L to learn effectively and autonomously in the real world without human intervention. Figure 3: This is the object repositioning task. The goal is to move the object from any starting configuration to a particular goal position and orientation. The system design outlined in Section 2, in principle, gives us a complete system to perform real world reinforcement learning without instrumentation. However, when ported to robotic problems in the real world, we find this basic design to be largely ineffective. To illustrate this, we present for a simulated robotic manipulation task that requires repositioning a free-floating objects with a three-fingered robotic hand, shown in Fig 3. We use this task for our investigative analysis, and show that the same insights extend to several other tasks (including real world tasks) in Section 6. The goal in this task is to reposition the object to a target pose from any initial pose in the arena. When the system is instantiated with vision-based SAC, rewards from goal images using VICE and run without episodic resets, we see that the algorithm fails to make progress (Fig 4). Although it might appear that this setup fits within the assumptions of all of the components that are used, the complete system is ineffective. Which particular components of this problem make it so difficult? To investigate this issue, we set up experiments that combine the three main ingredients: varying observation type (visual vs. low-dimensional state), reward structure (VICE vs. hand-defined rewards that utilize ground-truth object state) and the ability to reset (episodic resets vs. reset-free, non-episodic learning). We start by considering the training time reward under each combination of factors as shown in for the learned policy to achieve average training performance of less than 0.15 in pose distance (defined in App C.1.3) across 3 seeds. We compare training with true rewards vs. classifier-based rewards, with vs. without external resets, and from state vs. from vision on the object re-positioning task. We observe that learning without resets is more challenging than with resets and is much more challenging when combined with visual inputs. good training time reward with both vision and state while reset-free only obtains good training time reward with low-dimensional state; second, we find that the policy is able to pass the threshold for training time rewards in a non-episodic setting when operating from low-dimensional state, but not when using image observations. This suggests that combining the reset-free learning problem with visual observations makes it significantly more challenging than with low-dimensional state. However, the table in Fig 4 paints an incomplete picture. These numbers are related to the performance of the policies at training time, not how effective the learned policies are when being evaluated. When we consider the test-time performance (Fig 5) of the learned policies under reset free conditions, we obtain a different set of . While learning from low-dimensional state in the reset free setting is able to achieve decent training time reward, the test-time performance of the corresponding learned policies is very poor. This can likely be attributed to the fact that when the agent spends all its time stuck at the goal, it sees very little diversity of data in other parts of the state space, which significantly affects the efficacy of the actual policies being learned. This makes it very challenging to learn policies with completely reset-free schemes, which has prompted prior work to consider schemes such as learning reset-controllers . As we discuss in the following section and in our experiments, these schemes are often insufficient for learning effective policies in real world without any resets. These insights prompt us to propose simple solutions for instrumentation free reinforcement learning in Section 4. To address the challenges identified in Section 3, we present two improvements to the basic system outlined above, which we found to be essential for uninstrumented real-world training: randomized perturbation controllers and unsupervised representation learning. Incorporating these components into the system in Section 2 in a method that can learn in uninstrumented environments, as we will show in Section 6. From our observations in Fig 4, we can see that it is not effective to simply perform reset-free RL using a standard actor critic algorithm. Even the policies trained with full state information do not actually learn how to perform the desired repositioning task, as shown in Fig 5. This is because, once the policy has performed the task once, it does not need to perform it again, and therefore not only fails how to learn to perform the task reliably, but in fact tends to forget how to perform it at all. Prior work has considered addressing this problem by converting the reset-free learning problem into a more standard episodic problem, by learning a "reset controller," which is trained to reset the system to a particular initial state . However, as we will show in our experiments in Section 6, this still in policies that only succeed from a very narrow range of initial states. Indeed, prior reset controller methods all reset to a single initial state . We take a different approach to learning in a reset-free setting. Rather than attributing the problem to the variance of the initial state distribution, we hypothesize that a major problem with reset-free learning is that the support of the distribution of states visited by the policy is extremely narrow, which makes the learning problem challenging and doesn't allow the agent to learn how to perform the desired task from any state it might find itself in. In this view, the goal should not be to reduce the variance of the initial state distribution, but instead to increase it. To this end, we utilize what we call random perturbation controllers: controllers that introduce perturbations intermittently into the system through a policy that is trained to explore the environment. The standard actor π(a|s) is executed for H time-steps, following which we executed the perturbation controller π p (a|s) for H steps, and repeat. The policy π is trained with the VICE-based rewards for reaching the desired goals, while the perturbation controller π p is trained only with an intrinsic motivation objective that encourages visiting under-explored states. In our implementation, we use the random network distillation (RND) objective for training the perturbation controller . This procedure is described in detail in Appendix A, and is evaluated on the tasks we consider in Fig 6. The perturbation controller ensures that the support of the training distribution grows, and as a the policies can learn the desired behavior much more effectively, as shown in Fig 7. The perturbation controller discussed above allows us to learn policies that can succeed at the task from a variety of starting states. However, learning from visual observations still present a challenge. Our experiments in Fig 4 show that learning without resets from low-dimensional state is comparatively easier. We therefore aim to convert the vision-based learning problem into one that more closely resembles state-based learning, by training a variational autoencoder (VAE) and sharing the latent-variable representation across the actor and critic networks. Refer to Appendix B for more details. Note that we use a VAE as an instantiation of representation learning techniques that works well in the domains considered; approaches proposed by , and are also viable substitutes in this regard. While several prior works have also sought to incorporate unsupervised learning into reinforcement learning to make learning from images easier , we note that this becomes especially critical in the vision-based, reset-free setting, as motivated by the experiments in Section 3, which indicate that it is precisely this combination of factors -vision and no resetsthat presents the most difficult learning problem. Therefore, although the particular solution we use in our system has been studied in prior work, it is brought to bare to address a challenge that arises in real-world learning that we believe has not been explored in prior studies. These two improvements -the perturbation controller and joint training with an unsupervised learning loss, -combined with the general system described above, give us a complete practical system for real world reinforcement learning, which we term R3L. The overall method uses soft-actor critic for learning with visual observations and classifier based rewards with VICE, introduces auxiliary reconstruction objectives for unsupervised representation learning, and uses a perturbation controller during training to ensure that the learned policy can accomplish the task from a wide variety of states. Further details on the full system can be found in Appendix A. The primary contribution of this work is to propose a paradigm for continual instrumentation-free real world robotic learning, and a practical instantiations of such a system. Several prior works have applied policies learned with RL to particular tasks in the real world (; ; b). While many of these algorithms simply train in simulation and transfer ing policies to the real world, this paradigm is prone to domain shift and extensive simulation efforts (Sadeghi & Levine; ;). We instead focus on the paradigm of reinforcement learning purely in the real world. While algorithms have shown that we can indeed perform RL in the real world (; ; b;), these have been limited to highly instrumented laboratory settings. They have carefully hand-designed reward assignment schemes , reset mechanisms (; ; . In contrast, we are proposing to learn in environments with significantly less human instrumentation. This introduces a unique set of challenges not considered carefully in prior works. A key component of our system is learning from raw visual inputs. This has proven to be extremely difficult for policy gradient style algorithms (a) due to challenging representation learning problems. This has been made easier in simulated domains by using modified objectives such as auxiliary losses or by using more efficient algorithms (a). We show that reinforcement learning on raw visual input, while possible in standard RL settings, becomes significantly more challenging when considered in conjunction with nonepisodic, reset-free scenarios. Reward function design is crucial for any RL system, and is non-trivial to provide in the real world. Prior works have considered instrumenting the environment with additional sensors to evaluate rewards (; ;, which is a highly manual process, using demonstrations, which require manual effort to collect (; ;), or using interactive supervision from a user . In this work, we leverage the algorithm introduced by to assign rewards based on the likelihood of a goal classifier. While prior work also applied this method to robotic tasks , this was done in a setting where manual resets were provided, while we demonstrate that we can use learned rewards in a fully uninstrumented, reset-free setup. Learning without resets has been considered in prior works , although in different contexts -safe learning and learning compound controllers respectively. provide an algorithm to learn a reset controller with the goal of ensuring safe operation, but makes several assumptions that make it difficult to use in the real world: it assumes access to a ground truth reward function, it assumes access to an oracle function that can detect if an attempted reset by the reset policy was successful or not, and it assumes the ability to perform manual resets if the reset policy fails a certain number of times. In contrast, we propose an algorithm that allows for fully automated reinforcement learning in the real world. We compare to an ablation of our method that uses a reset controller similar to , and show that our method performs substantially better. Our perturbation controller also resembles the adversarial RL setup Pinto et al. (2017b);. However, while these prior methods explicitly aim to train policies that are robust to perturbations Pinto et al. (2017b) or explore effectively , we are concerned with learning without access to resets. While this line of work has connections to developmental robotics and it's subfields such as continual and lifelong learning, the goal of our work is to handle the practicalities of enabling reinforcement learning systems to learn in the real world without instrumentation or interruption, even for a single task setting, without multi-task considerations. The insights we make should also be applicable to improve developmental robotics algorithms. Though these settings don't touch on all aspects of developmental robotics, our work does relate with the subfields of continual learning , intrinsic motivation and sensory-motor development involving proprioceptive manipulation . In our experimental evaluation, we study how well the R3L system, described in Sections 2 and 4, can learn under realistic settings -visual observations, no hand-specified rewards, and no resets. We consider the following hypotheses: 1. Can we use R3L to learn complex robotic manipulation tasks without instrumentation? Does this system learn skills in both simulation and the real world? 2. Do the solutions proposed in Section 4 actually enable R3L to perform tasks without instrumentation that would not have been otherwise possible? We consider the task of dexterous manipulation with a three fingered robotic hand, called the ), on a number of simulated and real world environments. These tasks involve complex coordination of three fingers with 3 DoF each in order to manipulate objects. Prior works that used this robot utilized explicit resets and low-dimensional true state observations, while we consider settings with visual observations, no hand-specified rewards, and no resets. From left to right we depict valve rotation, bead manipulation and free object repositioning in simulation, as well as valve rotation and bead manipulation manipulation in the real world. The tasks in our experiments are shown in Fig 6: manipulating beads on an abacus row, valve rotation, and free object repositioning. These tasks represent a wide class of problems that robots might encounter in the real world. For each task, we consider the problem of reaching the depicted goal configuration: moving the abacus beads to a particular position, rotating the valve to a particular angle, and repositioning the free object to a particular goal position. For each task, policies are evaluated from a wide selection of initial configurations. Additional details about the tasks, and the evaluation procedures, including initial configurations and success metrics are provided in Appendix C. Videos and additional details can be found at https://sites.google.com/ view/realworld-rl/. We compare our entire proposed system implementation (Section 4) with a number of baselines and ablations. Importantly, all methods must operate under the same assumptions: no system instrumentation for state estimation, reward specification, or episodic resets. Firstly, we compare the performance of R3L to a system which uses SAC for vision-based RL from raw pixels, VICE for providing rewards and running reset-free (denoted as "VICE"). This corresponds to the vanilla version of R3L (Section 2), with none of the proposed insights and changes. We then compare with prior reset-free RL algorithms that explicitly learn a reset controller to alternate goals in the state space ("Reset Controller + VAE"). Lastly, we compare algorithm performance with two clear ablations -running R3L without the perturbation controller ("VICE + VAE") and without the unsupervised learning ("R3L w/o VAE"). This highlights the significance of all the components of R3L. From the experimental in Fig 7, it is clear that R3L is able to reach the best performance across tasks, while none of the other methods are able to solve all of the tasks. Different prior methods and ablations fail for different reasons: methods without the reconstruction objective struggle at parsing the high-dimensional input and are unable to solve the harder task; methods without the perturbation controller are ineffective at learning how to reach the goal from novel initialization positions for the more challenging object repositioning tasks, as discussed in Section 4. We note that an explicit reset controller, which can be viewed as a softer version of our perturbation controller with goal-directedness, learns to solve the easier tasks due to the reset controller encouraging exploration of the state space. In our experiments for free object repositioning, performance was reported across 3 choices of reset states. The high variance in evaluation performance indicates that the performance of such a controller (or a goal conditioned variant of it) is highly sensitive to the choice of reset states. A poor choice of reset states, such as two that are very close together, may yield poor exploration leading to performance similar to the single goal VICE baseline. Furthermore, the choice of reset states is highly task dependent and it is often not clear what choice of goals will yield the best performance. On the contrary, our method does not require such taskspecific knowledge and uses random perturbations to reset while training: this allows for a robust, instrumentation-free controller while also ensuring fast convergence. Since the aim of R3L is to enable uninstrumented training in the real world, we next evaluate our method on a real-world robotic system, providing evidence that our insights generalize to the real world without any instrumentation. In the instrumentation-free setting, we leave the robot unattended with a set of goal images provided and the algorithm learns the desired behavior through interaction. The experiments are performed on the DClaw robotic hand with an RGB camera as the only sensory input. Intermediate policies are saved at regular intervals, and evaluations of all policies are performed after training has completed. For valve rotation, we declare an evaluation rollout a success if the final orientation is within within 15 • of the goal. For bead manipulation, we declare success if all the beads are within 2cm of the goal state. Fig 8 compares the performance of our method without supervised learning ("R3L w/o VAE") in the real-world against a baseline that uses SAC for vision-based RL from raw pixels, VICE for providing rewards and running reset-free (denoted as "VICE"). We see that our method learns policies that evaluate from nearly all the initial configurations, whereas VICE fails to evaluate from most initial configurations. Fig 9 depicts sample evaluation rollouts of the policies learned using our method. For further details about real world experiments see Appendix C.2. We presented the design and instantiation of R3L, a system for real world reinforcement learning. We identify and investigate the various ingredients required for such a system to scale gracefully with minimal human engineering and supervision. We show that this system must be able to learn from raw sensory observations, learn from very easily specified reward functions without reward engineering, and learn without any episodic resets. We describe the basic elements that are required to construct such a system, and identify unexpected learning challenges that arise from interplay of these elements. We propose simple and scalable fixes to these challenges through introducing unsupervised representation learning and a randomized perturbation controller. We show the effectiveness on such a system at learning without instrumentation in several simulated and real world environments. The ability to train robots directly in the real world with minimal instrumentation opens a number of exciting avenues for future research. Robots that can learn unattended, without resets or handdesigned reward functions, can in principle collect very large amounts of experience autonomously, which may enable very broad generalization in the future. Furthermore, fully autonomous learning should make it possible for robots to acquire large behavioral repertoires, since each additional task requires only the initial examples needed to learn the reward. However, there are also a number of additional challenges, including sample complexity, optimization and exploration difficulties on more complex tasks, safe operation, communication latency, sensing and actuation noise, and so forth, all of which would need to be addressed in future work in order to enable truly scalable realworld robotic learning. Initialize RND target and predictor networks f (s),f (s) Initialize VICE reward classifier r VICE (s) Initialize replay buffer D Collect initial exploration data and add to D for i = 1 to N do 12: Sample an equal number of goal examples from G and negative examples from D for t = 1 to n VICE do 29: Train the VICE classifier on this batch with binary labels We use a variant of VICE which defines the reward as the logits of the classifier, notably omitting the −log(π(a|s)) term. We also regularize our classifier with mixup . We train all of our experiments using 200 goal images, which takes under an hour to collect in the real world for each task. We found it important to normalize the predictor errors, just as did. We train a standard beta-VAE to maximize the evidence lower bound, given by: To collect training data, we sampled random states in the observation space. In the real world, this sampling can be replaced with training an exploratory policy (i.e. using the RND reward as the policy's only objective). The learned weights of the encoder of the VAE are frozen, and the latent input is used to train the policy for reset-free RL. C TASK DETAILS C.1 SIMULATED TASKS We evaluated our system across three tasks in simulation: bead manipulation, valve rotation, free object repositioning. The bead manipulation task involves an abacus rod with four beads that can slide freely. The goal is to position two beads on each end from any initial configuration of beads. This can take the form of sliding one bead over (if three beads start on one side), two beads over (if all four beads start on one side), splitting beads apart (all four beads start in the middle), or some intermediate combination of those. The true reward is defined as the mean goal distance of all four beads. Policies are evaluated starting from the 8 initial configurations depicted in Fig 10. Evaluation performance reported in Section 6 for this task is defined as the final reward averaged across the 8 evaluation rollouts. The claw is positioned above a three pronged valve (15 cm in diameter). The objective is to turn the valve to a given orientation from any initial orientation. The "true reward" is r = − log(|θ state − θ goal |). Policies are evaluated starting from the 8 initial configurations depicted in Fig 11. Evaluation performance reported in Section 6 for this task is defined as the final orientation distance averaged across the 8 evaluation rollouts. The claw is positioned atop a free (6 DoF) three pronged object (15cm in diameter), which can translate and rotate within a 30cmx30cm box. For each setup we use an RGB camera to get images. We execute actions on the DClaw at 10Hz. In order to operate at such a high frequency while also training from images we sample and train asynchronously, but limit training to not exceed two gradient steps per transition sampled in the real world. Since direct performance metrics cannot be measured during training due to the lack of object instrumentation, evaluations of performance are done post-training. The task is identical to the one in simulation. Evaluations were done post-training. An evaluation trajectory was defined as a success if at the last step, the valve was within 15 degrees of the goal. Each policy was evaluated over 8 rollouts, with initial configurations evenly spaced out between at increments of 45 degrees. Results are reported in Figures 13, 14. The rod is 22cm in length, and each bead measures 3.5cm in diameter. Evaluations were done post-training, using the following procedure: the environment was manually reset to each of the 8 specified configurations shown in Figures 15 and 16 (which cover a full range of the state space) at the start of each evaluation rollout. An evaluation trajectory was defined as a success if at the last time step, all beads were within 2cm of their goal positions. Performance was evaluated at around 20 hours, at which point the policy achieved greater than 80% success on the 10 evaluation rollouts (a random policy achieved a success rate of 10%). Results are reported in Figs 15, 16. Under review as a conference paper at ICLR 2020 Figure 13: These are the of the evaluation rollouts on the valve rotation task in the real world using our method (without the VAE). Trained policies were saved at regular intervals and evaluated post-training. Each row is a different policy, and each column an evaluation rollout from a different initial configuration. The goal is highlighted in yellow. Our method is able to achieve high success rates after 5 hours of training. Under review as a conference paper at ICLR 2020 Figure 15: These are the of the evaluation rollouts on the valve rotation task in the real world using our method (without the VAE). Trained policies were saved at regular intervals and evaluated post-training. Each row is a different policy, and each column an evaluation rollout from a different initial configuration. The goal is highlighted in yellow. Our method is able to achieve high success rates after 17 hours of training. Figure 16: These are the of evaluation rollouts on the valve rotation task in the real world using the VICE single goal baseline. The policies fail to evaluate consistently, except when the initial configuration matches the goal configuration.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

System to learn robotic tasks in the real world with reinforcement learning without instrumentation

We address the task of autonomous exploration and navigation using spatial affordance maps that can be learned in a self-supervised manner, these outperform classic geometric baselines while being more sample efficient than contemporary RL algorithms


Question: represent the article for finding a single-sentence tldr

Question: We study the problem of cross-lingual voice conversion in non-parallel speech corpora and one-shot learning setting. Most prior work require either parallel speech corpora or enough amount of training data from a target speaker. However, we convert an arbitrary sentences of an arbitrary source speaker to target speaker's given only one target speaker training utterance. To achieve this, we formulate the problem as learning disentangled speaker-specific and context-specific representations and follow the idea of which uses Factorized Hierarchical Variational Autoencoder (FHVAE). After training FHVAE on multi-speaker training data, given arbitrary source and target speakers' utterance, we estimate those latent representations and then reconstruct the desired utterance of converted voice to that of target speaker. We use multi-language speech corpus to learn a universal model that works for all of the languages. We investigate the use of a one-hot language embedding to condition the model on the language of the utterance being queried and show the effectiveness of the approach. We conduct voice conversion experiments with varying size of training utterances and it was able to achieve reasonable performance with even just one training utterance. We also investigate the effect of using or not using the language conditioning. Furthermore, we visualize the embeddings of the different languages and sexes. Finally, in the subjective tests, for one language and cross-lingual voice conversion, our approach achieved moderately better or comparable compared to the baseline in speech quality and similarity. Variational Autoencoder proposed by Hsu et al BID0. Let a dataset D consist of N seq i. DISPLAYFORM0 2. Thus, joint probability with a sequence X i is: DISPLAYFORM1 This is illustrated in FIG0. For inference, we use variational inference to approximate the true 93 posterior and have: DISPLAYFORM2 Since sequence variational lower bound can be decomposed to segment variational lower bound, we 95 can use batches of segment instead of sequence level to maximize: are shown. In all subplots, the female and male embedding cluster locations are clearly separated. DISPLAYFORM3 Furthermore, the plot shows that the speaker embeddings of unique speakers fall near the same 147 location. One phenomenon that we notice is that the speaker embeddings for different languages and 148 gender fall to different locations for VAE-UNC, however, they fall closer to each other in VAE-CND. This might be due to the conditioning on language improving the representation ability of the model. Furthermore, we investigate the phonetic context embedding Z 1 for a sentence for four English test To subjectively evaluate voice conversion performance, we performed two perceptual tests. The To evaluate the speaker similarity of the converted utterances, we conducted a same-different speaker 181 similarity test. In this test, listeners heard two stimuli A and B with different content, and 182 were then asked to indicate whether they thought that A and B were spoken by the same, or by two 183 different speakers, using a five-point scale comprised of +2 (definitely same), +1 (probably same), 0

We use a Variational Autoencoder to separate style and content, and achieve voice conversion by modifying style embedding and decoding. We investigate using a multi-language speech corpus and investigate its effects.

Represent the short summary for finding the corresponding scientific article (long)

A residual EBM for text whose formulation is equivalent to discriminating between human and machine generated text. We study its generalization behavior.


Question: Represent the article for finding a single-sentence TLDR

Question: We propose an end-to-end-trainable attention module for convolutional neural network (CNN) architectures built for image classification. The module takes as input the 2D feature vector maps which form the intermediate representations of the input image at different stages in the CNN pipeline, and outputs a 2D matrix of scores for each map. Standard CNN architectures are modified through the incorporation of this module, and trained under the constraint that a convex combination of the intermediate 2D feature vectors, as parametrised by the score matrices, must alone be used for classification. Incentivised to amplify the relevant and suppress the irrelevant or misleading, the scores thus assume the role of attention values. Our experimental observations provide clear evidence to this effect: the learned attention maps neatly highlight the regions of interest while suppressing clutter. Consequently, the proposed function is able to bootstrap standard CNN architectures for the task of image classification, demonstrating superior generalisation over 6 unseen benchmark datasets. When binarised, our attention maps outperform other CNN-based attention maps, traditional saliency maps, and top object proposals for weakly supervised segmentation as demonstrated on the Object Discovery dataset. We also demonstrate improved robustness against the fast gradient sign method of adversarial attack. Feed-forward convolutional neural networks (CNNs) have demonstrated impressive on a wide variety of visual tasks, such as image classification, captioning, segmentation, and object detection. However, the visual reasoning which they implement in solving these problems remains largely inscrutable, impeding understanding of their successes and failures alike. One approach to visualising and interpreting the inner workings of CNNs is the attention map: a scalar matrix representing the relative importance of layer activations at different 2D spatial locations with respect to the target task BID21. This notion of a nonuniform spatial distribution of relevant features being used to form a task-specific representation, and the explicit scalar representation of their relative relevance, is what we term'attention'. Previous works have shown that for a classification CNN trained using image-level annotations alone, extracting the attention map provides a straightforward way of determining the location of the object of interest BID2 BID31 and/or its segmentation mask BID21, as well as helping to identify discriminative visual properties across classes BID31. More recently, it has also been shown that training smaller networks to mimic the attention maps of larger and higher-performing network architectures can lead to gains in classification accuracy of those smaller networks BID29.The works of BID21; BID2; BID31 represent one series of increasingly sophisticated techniques for estimating attention maps in classification CNNs. However, these approaches share a crucial limitation: all are implemented as post-hoc additions to fully trained networks. On the other hand, integrated attention mechanisms whose parameters are learned over the course of end-to-end training of the entire network have been proposed, and have shown benefits in various applications that can leverage attention as a cue. These include attribute prediction BID19, machine translation BID1, image captioning BID28 ) and visual question answering (VQA) BID24 BID26. Similarly to these approaches, we here represent attention as a probabilistic map over the input image locations, and implement its estimation via an end-to-end framework. The novelty of our contribution lies in repurposing the global image representation as a query to estimate multi-scale attention in classification, a task which, unlike e.g. image captioning or VQA, does not naturally involve a query. Fig. 1 provides an overview of the proposed method. Henceforth, we will use the terms'local features' and'global features' to refer to features extracted by some layer of the CNN whose effective receptive fields are, respectively, contiguous proper subsets of the image ('local') and the entire image ('global'). By defining a compatibility measure between local and global features, we redesign standard architectures such that they must classify the input image using only a weighted combination of local features, with the weights represented here by the attention map. The network is thus forced to learn a pattern of attention relevant to solving the task at hand. We experiment with applying the proposed attention mechanism to the popular CNN architectures of VGGNet BID20 and ResNet BID11, and capturing coarse-to-fine attention maps at multiple levels. We observe that the proposed mechanism can bootstrap baseline CNN architectures for the task of image classification: for example, adding attention to the VGG model offers an accuracy gain of 7% on CIFAR-100. Our use of attention-weighted representations leads to improved fine-grained recognition and superior generalisation on 6 benchmark datasets for domain-shifted classification. As observed on models trained for fine-grained bird recognition, attention aware models offer limited resistance to adversarial fooling at low and moderate L ∞ -noise norms. The trained attention maps outperform other CNN-derived attention maps BID31, traditional saliency maps BID14 BID30 ), and top object proposals on the task of weakly supervised segmentation of the Object Discovery dataset ). In §5, we present sample which suggest that these improvements may owe to the method's tendency to highlight the object of interest while suppressing clutter. Attention in CNNs is implemented using one of two main schemes -post hoc network analysis or trainable attention mechanisms. The former scheme has been predominantly employed to access network reasoning for the task of visual object recognition BID21 BID31 BID2. BID21 approximate CNNs as linear functions, interpreting the gradient of a class output score with respect to the input image as that class's spatial support in the image domain, i.e. its attention map. Importantly, they are one of the first to successfully demonstrate the use of attention for localising objects of interest using image-level category labels alone. BID31 apply the classifier weights learned for image-level descriptors to patch descriptors, and the ing class scores are used as a proxy for attention. Their improved localisation performance comes at the cost of classification accuracy. BID2 introduce attention in the form of binary nodes between the network layers of BID21. At test time, these attention maps are adapted to a fully trained network in an additional parameter tuning step. Notably, all of the above methods extract attention from fully trained CNN classification models, i.e. via post-processing. Subsequently, as discussed shortly, many methods have explored the per-formance advantages of optimising the weights of the attention unit in tandem with the original network weights. Trainable attention in CNNs falls under two main categories -hard (stochastic) and soft (deterministic). In the former, a hard decision is made regarding the use of an image region, often represented by a low-order parameterisation, for inference (; . The implementation is non-differentiable and relies on a sampling-based technique called REINFORCE for training, which makes optimising these models more difficult. On the other hand, the soft-attention method is probabilistic and thus amenable to training by backpropagation. The method of BID13 lies at the intersection of the above two categories. It uses a parameterised transform to estimate hard attention on the input image deterministically, where the parameters of the image transformation are estimated using differentiable functions. The soft-attention method of BID19 demonstrates improvements over the above by implementing nonuniform non-rigid attention maps which are better suited to natural object shapes seen in real images. It is this direction that we explore in our current work. Trainable soft attention in CNNs has mainly been deployed for query-based tasks BID1 BID19 BID28 ; BID24 BID26 . As is done with the exemplar captions of , the questions of BID24 ; BID26, and the source sentences of BID1, we here map an image to a high-dimensional representation which in turn highlights the relevant parts of the input image to guide the desired inference. We draw a close comparison to the progressive attention approach of Seo et al. FORMULA2 for attribute prediction. However, there are some noteworthy differences. Their method uses a one-hot encoding of category labels to query the image: this is unavailable to us and we hence substitute a learned representation of the global image. In addition, their sequential mechanism refines a single attention map along the length of the network pipeline. This doesn't allow for the expression of a complementary focus on different parts of the image at different scales as leveraged by our method, illustrated for the task of fine-grained recognition in §5.The applications of attention, in addition to facilitating the training task, are varied. The current work covers the following areas: · Domain shift: A traditional approach to handling domain shift in CNNs involves fine-tuning on the new dataset, which may require thousands of images from each target category for successful adaptation. We position ourselves amongst approaches that use attention BID31 BID13 to better handle domain changes, particularly those involving content, occlusion, and object pose variation, by selectively focusing to the objects of interest. · Weakly supervised semantic segmentation: This area investigates image segmentation using minimal annotations in the form of scribbles, bounding boxes, or image-level category labels. Our work uses category labels and is related to the soft-attention approach of BID12 . However, unlike the aformentioned, we do not explicitly train our model for the task of segmentation using any kind of pixel-level annotations. We evaluate the binarised spatial attention maps, learned as a by-product of training for image classification, for their ability to segment objects. · Adversarial robustness: The work by BID9 explores the ease of fooling deep classification networks by adding an imperceptible perturbation to the input image, implemented as an epsilon step in the direction opposite to the predicted class score gradient. The works by and BID8 argue that this vulnerability comes from relying on spurious or non-oracle features for classification. Consequently, BID8 demonstrate increased adversarial robustness by identifying and masking the likely adversarial feature dimensions. We experiment with performing such suppression in the spatial domain. The core goal of this work is to use attention maps to identify and exploit the effective spatial support of the visual information used by CNNs in making their classification decisions. This approach is premised on the hypothesis that there is benefit to identifying salient image regions and amplifying their influence, while likewise suppressing the irrelevant and potentially confusing information in other regions. In particular, we expect that enforcing a more focused and parsimonious use of image information should aid in generalisation over changes in the data distribution, as occurs for instance when training on one set and testing on another. Thus, we propose a trainable attention estimator and illustrate how to integrate it into standard CNN pipelines so as to influence their output as outlined DISPLAYFORM0 Figure 2: Attention introduced at 3 distinct layers of VGG. Lowest level attention maps appear to focus on the surroundings (i.e., the rocky mountain), intermediate level maps on object parts (i.e., harness and climbing equipment) and the highest level maps on the central object.above. The method is based on enforcing a notion of compatibility between local feature vectors extracted at intermediate stages in the CNN pipeline and the global feature vector normally fed to the linear classification layers at the end of the pipeline. We implement attention-aware classification by restricting the classifier to use only a collection of local feature vectors, as chosen and weighted by the compatibility scores, in classifying examples. We will first discuss the modification to the network architecture and the method of training it, given a choice of compatibility function. We will then conclude the method description by presenting alternate choices of the compatibility function. The proposed approach is illustrated in Fig. 2. Denote by DISPLAYFORM0 n } the set of feature vectors extracted at a given convolutional layer s ∈ {1, · · ·, S}. Here, each s i is the vector of output activations at the spatial location i of n total spatial locations in the layer. The global feature vector g has the entire input image as support and is output by the network's series of convolutional and nonlinear layers, having only to pass through the final fully connected layers to produce the original architecture's class score for that input. Assume for now the existence of a compatibility function C which takes two vectors of equal dimension as arguments and outputs a scalar compatibility score: this will be specified in 3.2.The method proceeds by computing, for each of one or more layers s, the set of compatibility scores C(L s, g) = {c DISPLAYFORM1 The normalised compatibility scores A s = {a In the case of a single layer (S = 1), the attention-incorporating global vector g a is computed as described above, then mapped onto a T -dimensional vector which is passed through a softmax layer to obtain class prediction probabilities {p 1,p 2, · · ·p T}, where T is the number of target classes. In the case of multiple layers (S > 1), we compare two options: concatenating the global vectors into a single vector DISPLAYFORM2 and using this as the input to the linear classification step as above, or, using S different linear classifiers and averaging the output class probabilities. All free network parameters are learned in end-to-end training under a cross-entropy loss function. The compatibility score function C can be defined in various ways. The alignment model from BID1; can be re-purposed as a compatibility function as follows: DISPLAYFORM0 Given the existing free parameters between the local and the global image descriptors in a CNN pipeline, we can simplify the concatenation of the two descriptors to an addition operation, without loss of generality. This allows us to limit the parameters of the attention unit. We then learn a single fully connected mapping from the ant descriptor to the compatibility scores. Here, the weight vector u can be interpreted as learning the universal set of features relevant to the object categories in the dataset. In that sense, the weights may be seen as learning the general concept of objectness. Alternatively, we can use the dot product between g and s i as a measure of their compatibility: DISPLAYFORM1 In this case, the relative magnitude of the scores would depend on the alignment between g and s iin the high-dimensional feature space and the strength of activation of s i. In a standard CNN architecture, a global image descriptor g is derived from the input image and passed through a fully connected layer to obtain class prediction probabilities. The network must express g via mapping of the input into a high-dimensional space in which salient higher-order visual concepts are represented by different dimensions, so as to render the classes linearly separable from one another. Our method encourages the filters earlier in the CNN pipeline to learn similar mappings, compatible with the one that produces g in the original architecture. This is achieved by allowing a local descriptor i of an image patch to contribute to the final classification step only in proportion to its compatibility with g as detailed above. That is, C(ˆ i, g) should be high if and only if the corresponding patch contains parts of the dominant image category. Note that this implies that the effective filters operating over image patches in the layers s must represent relatively'mature' features with respect to the classification goal. We thus expect to see the greatest benefit in deploying attention relatively late in the pipeline. Further, different kinds of class details are more easily accessible at different scales. Thus, in order to facilitate the learning of diverse and complementary attention-weighted features, we propose the use of attention over different spatial resolutions. The combination of the two factors stated above in our deploying the attention units after the convolutional blocks that are late in the pipeline, but before their corresponding maxpooling operations, i.e. before a drop in the spatial resolution. Note also that the use of the softmax function in normalising the compatibility scores enforces 0 ≤ a i ≤ 1 ∀i ∈ {1 · · · n} and i a i = 1, i.e. that the combination of feature vectors is convex. This ensures that features at different spatial locations must effectively compete against one another for their share of the attention map. The compatibility scores thus serve as a robust proxy for attention in the classification pipeline. To incorporate attention into the VGG network, we move each of the first 2 max-pooling layers of the baseline architecture after each of the 2 corresponding additional convolutional layers that we introduce at the end of the pipeline. By pushing the pooling operations further down the pipeline, we ensure that the local layers used for estimating attention have a higher resolution. Our modified model has 17 layers: 15 convolutional and 2 fully connected. The output activations of layer-16 (fc) define our global feature vector g. We use the local feature maps from layers 7, 10, and 13 (convolutional) for estimating attention. We compare our approach with the activation-based attention method of BID31, and the progressive attention mechanism of BID19. For RNs BID11, we use a 164-layered network. We replace the spatial average-pooling step after the computational block-4 with extra convolutional and max-pooling steps to obtain the global feature g. The outputs of blocks 2, 3, and 4 serve as the local descriptors for attention. For more details about network architectures refer to §A.2. Note that, for both of the above architectures, if the dimensionality of g and the local features of a layer s differ, we project g to the lower-dimensional space of the local features, instead of the reverse. This is done in order to limit the parameters at the classification stage. The global vector g, once mapped to a given dimensionality, is then shared by the local features from different layers s as long as they are of that dimensionality. We refer to the network Net with attention at the last level as Net-att, at the last two levels as Net-att2, and at the last three levels as Net-att3. We denote by dp the use of the dot product for matching the global and local descriptors and by pc the use of parametrised compatibility. We denote by concat the concatenation of descriptors from different levels for the final linear classification step. We use indep to denote the alternative of independent prediction of probabilities at different levels using separate linear classifiers: these probabilities are averaged to obtain a single score per class. We evaluate the benefits of incorporating attention into CNNs for the primary tasks of image classification and fine-grained object category recognition. We also examine robustness to the kind of adversarial attack discussed by BID9. Finally, we study the quality of attention maps as segmentations of image objects belonging to the network-predicted categories. For details of the datasets used and their pre-processing routines refer to §A.1, for network training schedules refer to §A.3, and for task-specific experimental setups refer to §A.4. Layer-13 att. Layer-10 att. Layer-13 att. Figure 3: Attention maps from VGG-att2 trained on lowres CIFAR-10 dataset focus sharply on the objects in high-res ImageNet images of CIFAR categories; contrasted here with the activation-based attention maps of BID29. Layer-10 att. Layer-13 att. Figure 4: VGG-att2 trained on CUB-200 for fine-grained bird recognition task: layer-10 learns to fixate on eye and beak regions, layer-13 on plumage and feet. Within the standard VGG architecture, the proposed attention mechanism provides noticeable performance improvement over baseline models (e.g. VGG, RN) and existing attention mechanisms (e.g. GAP, PAN) for visual recognition tasks, as seen in TAB2 & 2. Specifically, the VGG-att2-concat-pc model achieves a 2.5% and 7.4% improvement over baseline VGG for CIFAR-10 and CIFAR-100 classification, and 7.8% and 0.5% improvement for fine-grained recognition of CUB and SVHN categories. As is evident from Fig. 3, the attention mechanism enables the network to focus on the object of interest while suppressing the regions. For the task of fine-grained recognition, different layers learn specialised focus on different object parts as seen in Fig. 4. Note that the RN-34 model for CUB from TAB3 is pre-trained on ImageNet. In comparison, our networks are pre-trained using the much smaller and less diverse CIFAR-100. In spite of the low training cost, our networks are on par with the former in terms of accuracy. Importantly, despite the increase in the total network parameters due to the attention units, the proposed networks generalise exceedingly well to the test set. We are unable to compare directly with the CUB of BID13 due to a difference in dataset pre-processing. However, we improve over PAN BID19 ) by 4.5%, which has itself been shown to outperform the former at a similar task. For the remaining experiments, concat-pc is our implicit attention design unless specified otherwise. When the same attention mechanism is introduced into RNs we observe a marginal drop in performance: 0.9% on CIFAR-10 and 1.5% on CIFAR-100. It is possible that the skip-connections in RNs work in a manner similar to the proposed attention mechanism, i.e. by allowing select local features from earlier layers to skip through and influence inference. While this might make the performance improvement due to attention redundant, our method, unlike the skip-connections, is able to provide explicit attention maps that can be used for auxiliary tasks such as weakly supervised segmentation. Finally, the global feature vector is used as a query in our attention calculations. By changing the query vector, one could expect to affect the predicted attention pattern. A brief investigation of the extent to which the two compatibility functions allow for such post-hoc control is provided in §A.5. Figure 6: Network fooling rate measured as a percentage change in the predicted class labels w.r.t those predicted for the unperturbed images. From Fig. 6, the fooling rate of attention-aware VGG is 5% less than the baseline VGG at an L ∞ noise norm of 1. As the noise norm increases, the fooling rate saturates for the two networks and the performance gap gradually decreases. Interestingly, when the noise begins to be perceptible (see Fig. 5, col. 5), the fooling rate of VGG-att2 is around a percentage higher than that of VGG. Table 3: Cross-domain classification: Top-1 accuracies using models trained on CIFAR-10/100. CIFAR images cover a wider variety of natural object categories compared to those of SVHN and CUB. Hence, we use these to train different network architectures and use the networks as offthe-shelf feature extractors to evaluate their generalisability to new unseen datasets. From Table 3, attention-aware models consistently improve over the baseline models, with an average margin of 6%. We make two additional observations. Firstly, low-resolution CIFAR images contain useful visual properties that are transferrable to high-resolution images such as the 600 × 600 images of the Event-8 dataset . Secondly, training for diversity is better. CIFAR-10 and CIFAR-100 datasets contain the same corpus of images, only organised differently into 10 and 100 categories respectively. From the in Table 3, and the attention maps of Fig. 7, it appears that while learning to distinguish a larger set of categories the network is able to highlight more Layer-7 VGG-att3 (CIFAR10)Layer-10 VGG-att3 (CIFAR10)Layer-13 VGG-att3 (CIFAR10)Layer-7 VGG-att3 (CIFAR100)Layer-10 VGG-att3 (CIFAR100)Layer-13 VGG-att3 (CIFAR100)Level-3 RN-att2 (CIFAR100)Level-4 RN-att2 (CIFAR100)Figure 7: Row 1: Event-8 (croquet), Row 2: Scene-67 (bar). Attention maps for models trained on CIFAR-100 (c100) are more diverse than those from the models trained on CIFAR-10 (c10). Note the sharp attention maps in col. 7 versus the uniform ones in col. 4. Attention maps at lower levels appear to attend to part details (e.g. the stack of wine bottles in the bar (row 2)) and at a higher level on whole objects owing to a large effective receptive field. BID30 51.84 46.61 39.52 -Top object proposal-based -MCG BID0 32.02 54.21 37.85 -Joint segmentation-basedJoulin et al. 15.36 37.15 30.16 Object-discovery 55.81 64.42 51.65 Chen et al. BID3 54.62 69.20 44.46 Jain et al. BID6 58.65 66.47 53.57Figure 9: Jaccard scores (higher is better) for binarised attention maps from CIFAR-10/100 trained models tested on the Object Discovery dataset. From Table 9, the proposed attention maps perform significantly better at weakly supervised segmentation than those obtained using the existing attention methods BID31 BID19 and compare favourably to the top object proposal method, outperforming for all three categories by a minimum margin of 11% and 3% respectively. We do not compare with the CNN-based object proposal methods as they are trained using additional bounding box annotations. We surpass the saliency-based methods in the car category, but perform less well for the other two categories of airplane and horse. This could be due to the detailed structure and smaller size of objects of the latter two categories, see FIG3. Finally, we perform single-image inference and yet compare well to the joint inference methods using a group of test images for segmenting the common object category. We propose a trainable attention module for generating probabilistic landscapes that highlight where and in what proportion a network attends to different regions of the input image for the task of classification. We demonstrate that the method, when deployed at multiple levels within a network, affords significant performance gains in classification of seen and unseen categories by focusing on the object of interest. We also show that the attention landscapes can facilitate weakly supervised segmentation of the predominant object. Further, the proposed attention scheme is amenable to popular post-processing techniques such as conditional random fields for refining the segmentation masks, and has shown promise in learning robustness to certain kinds of adversarial attacks. We evaluate the proposed attention models on CIFAR-10 , CIFAR-100 (, SVHN) and CUB-200-2011 for the task of classification. We use the attention-incorporating VGG model trained on CUB-200-2011 for investigating robustness to adversarial attacks. For cross-domain classification, we test on 6 standard benchmarks including STL BID31 ), Caltech-256 (and Action-40 . We use the Object Discovery dataset for evaluating weakly supervised segmentation performance. A detailed summary of these datasets can be found in Table 4 .For all datasets except SVHN and CUB, we perform mean and standard deviation normalisation as well as colour normalisation. For CUB-200-2011, the images are cropped using ground-truth bounding box annotations and resized. For cross-domain image classification we downsample the input images to avoid the memory overhead. Dataset Size (total/train/test/extra) Number of classes / Type Resolution Tasks Table 4: Summary of datasets used for experiments across different tasks (C: classification, C-c: classification cross-domain, S: segmentation).The natural images span from those of objects in plain to cluttered indoor and outdoor scenes. The objects vary from simple digits to humans involved in complex activities. Progressive attention networks: We experiment with the progressive attention mechanism proposed by BID19 as part of our 2-level attention-based VGG models. The attention at the lower level (layer-10) is implemented using a 2D map of compatibility scores, obtained using the parameterised compatibility function discussed in §3.2. Note that at this level, the compatibility scores are not jointly normalised using the softmax operation, but are normalised independently using the pointwise sigmoid function. These scores, at each spatial location, are used to weigh the corresponding local features before the feature vectors are fed to the next network layer. This is the filtering operation at the core of the progressive attention scheme proposed by BID19. For the final level, attention is implemented in the same way as in VGG-att2-concat-pc. The compatibility scores are normalised using a softmax operation and the local features, added in proportion to the normalised attention scores, are trained for image classification. We implement and evaluate the above-discussed progressive attention approach as well as the proposed attention mechanism with the VGG architecture using the codebase provided here: https://github.com/szagoruyko/cifar.torch. The code for CIFAR dataset normalisation is included in the repository. For the ResNet architecture, we make the attention-related modifications to the network specification provided here: https://github.com/szagoruyko/wide-residualnetworks/tree/fp16/models. The baseline ResNet implementation consists of 4 distinct levels that project the RGB input onto a 256-dimensional space through 16-, 64-, and 128-dimensional embedding spaces respectively. Each level excepting the first, which contains 2 convolutional layers separated by a non-linearity, contains n-residual blocks. Each residual block in turn contains a maximum of 3 convolutional layers interleaved with non-linearities. This yields a network definition of 9n + 2 parameterised layers BID11. We work with an n of 18 for a 164-layered network. Batch normalization is incorporated in a manner similar to other contemporary networks. We replace the spatial average pooling layer after the final and 4 th level by convolutional and maxpooling operations which gives us our global feature vector g. We refer to the network implementing attention at the output of the last level as RN-att and with attention at the output of last two levels as RN-att2. Following the obtained on the VGG network, we train the attention units in the concat-pc framework. A.3 TRAINING ROUTINES VGG networks for CIFAR-10, CIFAR-100 and SVHN are trained from scratch. We use a stochastic gradient descent (SGD) optimiser with a batch size of 128, learning rate decay of 10 −7, weight decay of 5 × 10 −4, and momentum of 0.9. The initial learning rate for CIFAR experiments is 1 and for SVHN is 0.1. The learning rate is scaled by 0.5 every 25 epochs and we train over 300 epochs for convergence. For CUB, since the training data is limited, we initialise the model with the weights learned for CIFAR-100. We use the transfer-learning training schedule inspired by BID17. Thus the training starts at a learning rate of 0.1 for first 30 epochs, is multiplied by 2 twice over the next 60 epochs, and then scaled by 0.5 every 30 epochs for the next 200 epochs. For ResNet, the networks are trained using an SGD optimizer with a batch size of 64, initial learning rate of 0.1, weight decay of 5 × 10 −4, and a momentum of 0.9. The learning rate is multiplied by 0.2 after 60, 120 and 160 epochs. The network is trained for 200 epochs until convergence. We train the models for CIFAR-10 and CIFAR-100 from scratch. All models are implemented in Torch and trained with an NVIDIA Titan-X GPU. Training takes around one to two days depending on the model and datasets. We generate the adversarial images using the fast gradient sign method of BID9 and observe the network fooling behaviour at increasing L ∞ norms of the perturbations. For cross-domain classification, we extract features at the input of the final fully connected layer of each model, use these to train a linear SVM with C = 1 and report the of a 5-fold cross validation, similar to the setup used by BID31. At no point do we finetune the networks on the target datasets. We perform ZCA whitening on all the evaluation datasets using the pre-processing Python scripts specified in the following for whitening CIFAR datasets: https://github.com/szagoruyko/wide-residual-networks/tree/fp16.For weakly supervised segmentation, the evaluation datasets are preprocessed for colour normalisation using the same scheme as adopted for normalising the training datasets of the respective models. For the proposed attention mechanism, we combine the attention maps from the last 2 levels using element-wise multiplication, take the square root of the to re-interpret it as a probability distribution (in the limit of the two probabilistic attention distributions approaching each other), rescale the values in the ant map to a range of and binarise the map using the Otsu binarization threshold. For the progressive attention mechanism of Seo et al. FORMULA2, we simply multiply the attention maps from the two different levels without taking their square root, given that these attention maps are not complementary but sequential maps used to fully develop the final attention distribution. The rest of the operations of magnitude rescaling and binarisation are commonly applied to all the final attention maps, including those obtained using GAP BID31. In our framework, the global feature vector is used as a query for estimating attention on the local image regions. Thus, by changing the global feature vector, one could expect to affect the attention distribution estimated over the local regions in a predictable manner. The extent to which the two different compatibility functions, the parameterised and the dot-product-based compatibility functions, allow for such external control over the estimated attention patterns may be varied. For the purpose of the analysis, we consider two different network architectures. The first is the (VGG-att2)-concat-dp (DP) model from TAB2 which uses a dot-product-based compatibility function. The other is the (VGG-att3)-concat-pc (PC) model which makes use of the parameterised compatibility function. In terms of the dataset, we make use of the extra cosegmentation image set available with the Object Discovery dataset package. We select a single image centrally focused on an instance of a given object category and call this the query image. We then gather a few distinct images that contain objects of the same category but in a more cluttered environment with pose and intra-class variations. We call these the target images. In order to visualise the role of the global feature vector in driving the estimated attention patterns, we perform two rounds of experiments. In the first round, we obtain both the global and local image feature vectors from a given target image, shown in column 2 of every row of FIG0. The processing follows the standard protocol and the ing attention patterns at layer 10 for the two architectures can be seen in columns 3 and 6 of the same figure. In the second round, we obtain the local feature vectors from the target image but the global feature vector is obtained by processing the query image specific to the category being considered, shown in column 1. The new attention patterns are displayed in columns 4 and 7 respectively. The changes in the attention values at different spatial locations as a proportion of the original attention pattern values are shown in columns 5 and 8 respectively. Notably, for the dot-product-based attention mechanism, the global vector plays a prominent role in guiding attention. This is visible in the increase in the attention magnitudes at the spatial locations near or related to the query image object. On the other hand, for the attention mechanism that makes use of the parameterised compatibility function, the global feature vector seems to be redundant. Any change in the global feature vector does not transfer to the ing attention map. In fact, numerical observations show that the magnitudes of the global features are often a couple of orders of magnitude smaller than those of the corresponding local features. Thus, a change in the global feature vector has little to no impact on the predicted attention scores. Yet, the attention maps themselves are able to consistently highlight object-relevant image regions. Thus, it appears that in the case of parameterised compatibility based attention, the object-centric high-order features are learned as part of the weight vector u. These features are adapted to the training dataset and are able to generalise to new images inasmuch as the object categories at test time are similar to those seen during training.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

The paper proposes a method for forcing CNNs to leverage spatial attention in learning more object-centric representations that perform better in various respects.

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Recurrent neural network(RNN) is an effective neural network in solving very complex supervised and unsupervised tasks. There has been a significant improvement in RNN field such as natural language processing, speech processing, computer vision and other multiple domains. This paper deals with RNN application on different use cases like Incident Detection, Fraud Detection, and Android Malware Classification. The best performing neural network architecture is chosen by conducting different chain of experiments for different network parameters and structures. The network is run up to 1000 epochs with learning rate set in the range of 0.01 to 0.5.Obviously, RNN performed very well when compared to classical machine learning algorithms. This is mainly possible because RNNs implicitly extracts the underlying features and also identifies the characteristics of the data. This lead to better accuracy. In today's data world, malware is the common threat to everyone from big organizations to common people and we need to safeguard our systems, computer networks, and valuable data. Cyber-crimes has risen to the peak and many hacks, data stealing, and many more cyber-attacks. Hackers gain access through any loopholes and steal all valuable data, passwords and other useful information. Mainly in android platform malicious attacks increased due to increase in large number of application. In other hand its very easy for persons to develop multiple malicious malwares and feed it into android market very easily using a third party software's. Attacks can be through any means like e-mails, exe files, software, etc. Criminals make use of security vulnerabilities and exploit their opponents. This forces the importance of an effective system to handle the fraudulent activities. But today's sophisticated attacking algorithms avoid being detected by the security Email address: harishharunn@gmail.com (Mohammed Harun Babu R) mechanisms. Every day the attackers develop new exploitation techniques and escape from Anti-virus and Malware softwares. Thus nowadays security solution companies are moving towards deep learning and machine learning techniques where the algorithm learns the underlying information from the large collection of security data itself and makes predictions on new data. This, in turn, motivates the hackers to develop new methods to escape from the detection mechanisms. Malware attack remains one of the major security threat in cyberspace. It is an unwanted program which makes the system behave differently than it is supposed to behave. The solutions provided by antivirus software against this malware can only be used as a primary weapon of resistance because they fail to detect the new and upcoming malware created using polymorphic, metamorphic, domain flux and IP flux. The machine learning algorithms were employed which solves complex security threats in more than three decades BID0. These methods have the capability to detect new malwares. Research is going at a high phase for security problems like Intrusion Detection Systems(IDS), Mal-ware Detection, Information Leakage, etc. Fortunately, today's Deep Learning(DL) approaches have performed well in various long-standing AI challenges BID1 such as nlp, computer vision, speech recognition. Recently, the application of deep learning techniques have been applied for various use cases of cyber security BID2.It has the ability to detect the cyber attacks by learning the complex underlying structure, hidden sequential relationships and hierarchical feature representations from a huge set of security data. In this paper, we are evaluating the efficiency of SVM and RNN machine learning algorithms for cybersecurity problems. Cybersecurity provides a set of actions to safeguard computer networks, systems, and data. This paper is arranged accordingly where related work are discussed in section 2 the knowledge of recurrent neural network (RNN) in section 3.In section 4 proposed methodology including description,data set are discussed and at last are furnished in Section 5. Section 6 is conclude with . In this section related work for cybersecurity use cases is discussed: Android Malware Classification (T1), Incident Detection (T2), and Fraud Detection (T3). The most commonly used approach for Malware detection in Android devices is the static and dynamic approach BID3. In the static approach, all the android permissions are collected by unpacking the application and whereas, in dynamic approach, the run-time execution attributes like system calls, network connections, electricity, user interactions and efficient utilization of memory. Most of the commercial systems used today use both the static and dynamic approach. For low computational cost, resource utilization, time resource Static analysis is mainly preferred for Android devices. Meanwhile dynamic analysis has the advantage to detect metamorphic and polymorphic malware. BID4 have evaluated the performance of traditional ML algorithms for malware detection on Android devices without using the API calls and permission as features. MalDozer proposed the use of API calls with deep learning approach to detect the Android malware and classify them accordingly BID5. BID6 API calls contains schematic information which helps in understand the intention of the app indirectly without any user interface. Using embedding techniques at training phase API calls are extracted using DEX assembly BID5 which helps in effective malware detection on neural networks. The security issues in cloud computing are briefly discussed in BID7. BID8 proposed ML-based anomaly detection that acts on the network, service and work-flow layers. A hybrid of both machine learning and rulebased systems are combined for intrusion detection in the cloud infrastructure BID9. BID10 shows how Incident Detection can perform well than intrusion detection. In BID11 discusses a detailed study on 6 different traditional ML classifiers in finding the credit card frauds, financial frauds. Credit card frauds are detected using Convolution Neural Networks. Fraud Detection in crowd sourcing projects is discussed in BID12.Statistical Fraud Detection method model is trained to discriminate the fraudulent and non fraudulent using supervised and unsupervised methods in credit card frauds. BID6 Especially in communication networks Fraud Detection are rectified using supervised learning by statistical learning of behaviour of networks us using Bayesian network approach. Data mining approaches related to financial Fraud Detection are discussed in BID13. BID14 mainly discusses the Fraud Detection in today's new Online e-commerce transaction using Recurrent Neural Network(RNN) which performed very well. Based on this a detailed survey is conducted in BID15. The risks and trust involved in e-commerce market are detailed studied in BID16. The first task is an android classification task. The dataset is created from a set of APK packages files collected from the Opera Mobile Store from Jan to Sep 2014 is used. This dataset consists of API(Application Programming Interface) information for 61,730 APK files where 30,897 files for training and 30,833 files for testing BID17. The second task is incident detection. This dataset contains operational log file that was captured from Unified Threat Management (UTM) of UniteCloud. Task 3 is Fraud Detection. This dataset is anonymised data that was unified using the highly correlated rule based uniformly distributed synthetic data (HCRUD) approach by considering similar distribution of features. To find an optimal , three trails of experiment with 700 epochs has run with learning rate varying in the range [0.01-0.5]. The highest 10-fold cross-validation accuracy was obtained by using the learning rate of 0.01. There was a sudden decrease in accuracy at learning rate 0.05 and finally attained highest accuracy at learning rates of 0.035, 0.045 and 0.05 in comparison to learning rate 0.01. This accuracy may have been enhanced by running the experiments till 1000 epochs. As more complex architectures we have experimented with, showed less performance within 500 epochs, so 0.01 as learning rate for the rest of the experiments by taking training time and computational cost into account. The RNN 1 to 6 layer network topology are used in order to find an optimum network structure for our input data since we don't know the optimal number of layers and neurons. We run 3 trails of experiments for each RNN network toplogy. Each trail of the experiment was run till 700 epochs. It was observed that most of the deep learning architectures learn the normal category patterns of input data within 400 epochs itself. The number of epochs required to learn the malicious category data usually varies. This complex architecture networks required a large number of iterations in order to reach the best accuracy. At last, we obtained the bestperformed network topology for each use case. For Task Two and Task Three, 3 layer RNN network performed well. For Task One, the 6 layer RNN network gave a good performance in comparison to the 4 layer RNN. Then we decided to use 6 layer RNN network for the rest of the experiments. 10-fold cross-validation accuracy of each RNN network topology for all use cases is shown in TAB2. An intuitive overview of our proposed RNN architecture for all use cases is shown in FIG0 This consists of the input layer with six hidden layers and an output layer. An input layer contains 4896 neurons for Task One, 9 neurons for Task Two and 12 neurons for Task Three. An output layer contains 2 neurons for Task One, 3 neurons for Task Two and 2 neurons for Task Three. The detailed structure and configuration of proposed RNN architecture are shown in TAB2. The neurons in input to hidden layer and hidden to output layer are fully connected. The proposed Recurrent Network is composed of recurrent layers, fully-connected layers, batch normalization layers and dropout layers. It contains the recurrent units/neurons. The units have self-connection/loops. This helps to carry out the previous time step information for the future time step. Batch Normalization and Regularization: To obviate overfitting and speed up the RNN model training, Dropout (0.001) BID18 and Batch Normalization BID19 was used in between fully-connected layers. A dropout removes neurons with their connections randomly. In our alternative architectures for Task 1, the recurrent networks could easily overfit the training data without regularization even when trained on large number samples. Classification: For classification, the final fully connected layer follows sigmoid activation function for Task One and Task Two, softmax for Task Three. The fully connected layer absorb the non-linear kernel and sigmoid layer output zero (benign) and output one (malicious), softmax provides the probability score for each class. The prediction loss for Task One and Task Two is estimated using binary cross entropy DISPLAYFORM0 where vector predicted probability is denoted by pd testing data set, ed is a vector of the expected class label, values are either 0 or 1.The prediction loss for Task Three is estimated using categorical-cross entropy DISPLAYFORM1 where pd true probability distribution, ed is predicted probability distribution. We have used sgd as an optimizer to minimize the loss of binary-cross entropy and categorical-cross entropy. We have evaluated the proposed RNN model against classical machine learning classifier SVM, on 3 different cybersecurity use cases.1.Identifying Android malware based on API information, 2.Incident Detection over unified threat management (UTM) operation on Unite Cloud, 3.Fraud Detection in financial transactions. The detailed of proposed RNN model on 3 different use cases are displayed in TAB3. In this paper performance of RNN Vs other classical machine learning classifiers are evaluated for cybersecuriy use cases such as Android malware classification, incident detection, and fraud detection. In all the three

Recurrent neural networks for Cybersecurity use-cases

Represent the short summary for finding the corresponding scientific article (long).

Data stream algorithms can be improved using deep learning, while retaining performance guarantees.