Supervised learning, more specifically Convolutional Neural Networks (CNN), has surpassed human ability in some visual recognition tasks such as detection of traffic signs, faces and handwritten numbers. On the other hand, even state-of-the-art reinforcement learning (RL) methods have difficulties in environments with sparse and binary rewards. They requires manually shaping reward functions, which might be challenging to come up with. These tasks, however, are trivial to human. One of the reasons that human are better learners in these tasks is that we are embedded with much prior knowledge of the world. These knowledge might be either embedded in our genes or learned from imitation - a type of supervised learning. For that reason, the best way to narrow the gap between machine and human learning ability should be to mimic how we learn so well in various tasks by a combination of RL and supervised learning. Our method, which integrates Deep Deterministic Policy Gradients and Hindsight Experience Replay (RL method specifically dealing with sparse rewards) with an experience ranking CNN, provides a significant speedup over the learning curve on simulated robotics tasks. Experience ranking allows high-reward transitions to be replayed more frequently, and therefore help learn more efficiently. Our proposed approach can also speed up learning in any other tasks that provide additional information for experience ranking.
@cite_19: We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.
@cite_18: An artificial agent is developed that learns to play a diverse range of classic Atari 2600 computer games directly from sensory experience, achieving a performance comparable to that of an expert human player; this work paves the way to building general-purpose learning algorithms that bridge the divide between perception and action.
@cite_16: SUMMARY The acquisition of new memories for places and events requires synaptic plasticity in the hippocampus,andplasticitydependsontemporalcoordination among neurons. Spatial activity in the hippocampus is relatively disorganizedduring the initial exploration of a novel environment, however, and it is unclear howneuralactivityduringtheinitialstagesoflearning drives synaptic plasticity. Here we show that pairs of CA1 cells that represent overlapping novel locations are initially more coactive and more precisely coordinatedthan arecells representing overlappingfamiliar locations. This increased coordination occurrs specifically during brief, high-frequency events (HFEs) in the local field potential that are similar to ripples and is not associated with better coordination of place-specific neural activity outside of HFEs. As novel locations become more familiar, correlations betweencellpairsdecrease.Thus,hippocampalneural activity during learning has a unique structure that iswellsuitedtoinducesynapticplasticityandtoallow for rapid storage of new memories.
@cite_20: The consolidation of memories for places and events is thought to rely, at the network level, on the replay of spatially tuned neuronal firing patterns representing discrete places and spatial trajectories. This occurs in the hippocampal-entorhinal circuit during sharp wave ripple events (SWRs) that occur during sleep or rest. Here, we review theoretical models of lingering place cell excitability and behaviorally induced synaptic plasticity within cell assemblies to explain which sequences or places are replayed. We further provide new insights into how fluctuations in cholinergic tone during different behavioral states might shape the direction of replay and how dopaminergic release in response to novelty or reward can modulate which cell assemblies are replayed.
However, the current way to use experience replay is to uniformly sampled from first-in-first-out (FIFO) buffers of experiences @cite_19 @cite_18 . By doing this way, the importance of each transition is neglected, and we might replay experience that is not very helpful for the learning at all. When faced with the design choice of which experiences to restore, neuroscience research show that surprising @cite_16 and rewarding experiences are more preferred @cite_20 .