# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Project: Train a Quadcopter How to Fly
#
# Design an agent to fly a quadcopter, and then train it using a reinforcement learning algorithm of your choice! 
#
# Try to apply the techniques you have learnt, but also feel free to come up with innovative ideas and test them.

# ## Instructions
#
# Take a look at the files in the directory to better understand the structure of the project. 
#
# - `task.py`: Define your task (environment) in this file.
# - `agents/`: Folder containing reinforcement learning agents.
#     - `policy_search.py`: A sample agent has been provided here.
#     - `agent.py`: Develop your agent here.
# - `physics_sim.py`: This file contains the simulator for the quadcopter.  **DO NOT MODIFY THIS FILE**.
#
# For this project, you will define your own task in `task.py`.  Although we have provided a example task to get you started, you are encouraged to change it.  Later in this notebook, you will learn more about how to amend this file.
#
# You will also design a reinforcement learning agent in `agent.py` to complete your chosen task.  
#
# You are welcome to create any additional files to help you to organize your code.  For instance, you may find it useful to define a `model.py` file defining any needed neural network architectures.
#
# ## Controlling the Quadcopter
#
# We provide a sample agent in the code cell below to show you how to use the sim to control the quadcopter.  This agent is even simpler than the sample agent that you'll examine (in `agents/policy_search.py`) later in this notebook!
#
# The agent controls the quadcopter by setting the revolutions per second on each of its four rotors.  The provided agent in the `Basic_Agent` class below always selects a random action for each of the four rotors.  These four speeds are returned by the `act` method as a list of four floating-point numbers.  
#
# For this project, the agent that you will implement in `agents/agent.py` will have a far more intelligent method for selecting actions!

# +
import random

class Basic_Agent():
    def __init__(self, task):
        self.task = task
    
    def act(self):
        new_thrust = random.gauss(450., 25.)
        return [new_thrust + random.gauss(0., 1.) for x in range(4)]


# -

# Run the code cell below to have the agent select actions to control the quadcopter.  
#
# Feel free to change the provided values of `runtime`, `init_pose`, `init_velocities`, and `init_angle_velocities` below to change the starting conditions of the quadcopter.
#
# The `labels` list below annotates statistics that are saved while running the simulation.  All of this information is saved in a text file `data.txt` and stored in the dictionary `results`.  

# +
# %load_ext autoreload
# %autoreload 2

import csv
import numpy as np
from tasks import ExampleTask

# Modify the values below to give the quadcopter a different starting position.
runtime = 5.                                     # time limit of the episode
init_pose = np.array([0., 0., 10., 0., 0., 0.])  # initial pose
init_velocities = np.array([0., 0., 0.])         # initial velocities
init_angle_velocities = np.array([0., 0., 0.])   # initial angle velocities
file_output = 'data.txt'                         # file name for saved results

# Setup
task = ExampleTask(init_pose, init_velocities, init_angle_velocities, runtime)
agent = Basic_Agent(task)
done = False
labels = ['time', 'x', 'y', 'z', 'phi', 'theta', 'psi', 'x_velocity',
          'y_velocity', 'z_velocity', 'phi_velocity', 'theta_velocity',
          'psi_velocity', 'rotor_speed1', 'rotor_speed2', 'rotor_speed3', 'rotor_speed4']
results = {x : [] for x in labels}

# Run the simulation, and save the results.
with open(file_output, 'w') as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(labels)
    while True:
        rotor_speeds = agent.act()
        _, _, done = task.step(rotor_speeds)
        to_write = [task.sim.time] + list(task.sim.pose) + list(task.sim.v) + list(task.sim.angular_v) + list(rotor_speeds)
        for ii in range(len(labels)):
            results[labels[ii]].append(to_write[ii])
        writer.writerow(to_write)
        if done:
            break
# -

# Run the code cell below to visualize how the position of the quadcopter evolved during the simulation.

# +
import matplotlib.pyplot as plt
# %matplotlib inline

plt.plot(results['time'], results['x'], label='x')
plt.plot(results['time'], results['y'], label='y')
plt.plot(results['time'], results['z'], label='z')
plt.legend()
_ = plt.ylim()
# -

# The next code cell visualizes the velocity of the quadcopter.

plt.plot(results['time'], results['x_velocity'], label='x_hat')
plt.plot(results['time'], results['y_velocity'], label='y_hat')
plt.plot(results['time'], results['z_velocity'], label='z_hat')
plt.legend()
_ = plt.ylim()

# Next, you can plot the Euler angles (the rotation of the quadcopter over the $x$-, $y$-, and $z$-axes),

plt.plot(results['time'], results['phi'], label='phi')
plt.plot(results['time'], results['theta'], label='theta')
plt.plot(results['time'], results['psi'], label='psi')
plt.legend()
_ = plt.ylim()

# before plotting the velocities (in radians per second) corresponding to each of the Euler angles.

plt.plot(results['time'], results['phi_velocity'], label='phi_velocity')
plt.plot(results['time'], results['theta_velocity'], label='theta_velocity')
plt.plot(results['time'], results['psi_velocity'], label='psi_velocity')
plt.legend()
_ = plt.ylim()

# Finally, you can use the code cell below to print the agent's choice of actions.  

plt.plot(results['time'], results['rotor_speed1'], label='Rotor 1 revolutions / second')
plt.plot(results['time'], results['rotor_speed2'], label='Rotor 2 revolutions / second')
plt.plot(results['time'], results['rotor_speed3'], label='Rotor 3 revolutions / second')
plt.plot(results['time'], results['rotor_speed4'], label='Rotor 4 revolutions / second')
plt.legend()
_ = plt.ylim()

# When specifying a task, you will derive the environment state from the simulator.  Run the code cell below to print the values of the following variables at the end of the simulation:
# - `task.sim.pose` (the position of the quadcopter in ($x,y,z$) dimensions and the Euler angles),
# - `task.sim.v` (the velocity of the quadcopter in ($x,y,z$) dimensions), and
# - `task.sim.angular_v` (radians/second for each of the three Euler angles).

# the pose, velocity, and angular velocity of the quadcopter at the end of the episode
print(task.sim.pose)
print(task.sim.v)
print(task.sim.angular_v)

# In the sample task in `task.py`, we use the 6-dimensional pose of the quadcopter to construct the state of the environment at each timestep.  However, when amending the task for your purposes, you are welcome to expand the size of the state vector by including the velocity information.  You can use any combination of the pose, velocity, and angular velocity - feel free to tinker here, and construct the state to suit your task.
#
# ## The Task
#
# A sample task has been provided for you in `task.py`.  Open this file in a new window now. 
#
# The `__init__()` method is used to initialize several variables that are needed to specify the task.  
# - The simulator is initialized as an instance of the `PhysicsSim` class (from `physics_sim.py`).  
# - Inspired by the methodology in the original DDPG paper, we make use of action repeats.  For each timestep of the agent, we step the simulation `action_repeats` timesteps.  If you are not familiar with action repeats, please read the **Results** section in [the DDPG paper](https://arxiv.org/abs/1509.02971).
# - We set the number of elements in the state vector.  For the sample task, we only work with the 6-dimensional pose information.  To set the size of the state (`state_size`), we must take action repeats into account.  
# - The environment will always have a 4-dimensional action space, with one entry for each rotor (`action_size=4`). You can set the minimum (`action_low`) and maximum (`action_high`) values of each entry here.
# - The sample task in this provided file is for the agent to reach a target position.  We specify that target position as a variable.
#
# The `reset()` method resets the simulator.  The agent should call this method every time the episode ends.  You can see an example of this in the code cell below.
#
# The `step()` method is perhaps the most important.  It accepts the agent's choice of action `rotor_speeds`, which is used to prepare the next state to pass on to the agent.  Then, the reward is computed from `get_reward()`.  The episode is considered done if the time limit has been exceeded, or the quadcopter has travelled outside of the bounds of the simulation.
#
# In the next section, you will learn how to test the performance of an agent on this task.

# ## The Agent
#
# The sample agent given in `agents/policy_search.py` uses a very simplistic linear policy to directly compute the action vector as a dot product of the state vector and a matrix of weights. Then, it randomly perturbs the parameters by adding some Gaussian noise, to produce a different policy. Based on the average reward obtained in each episode (`score`), it keeps track of the best set of parameters found so far, how the score is changing, and accordingly tweaks a scaling factor to widen or tighten the noise.
#
# Run the code cell below to see how the agent performs on the sample task.

# +
import sys
import pandas as pd
from agents import PolicySearchAgent
from tasks import ExampleTask

num_episodes = 1000
target_pos = np.array([0., 0., 10.])
task = ExampleTask(target_pos=target_pos)
agent = PolicySearchAgent(task) 

for i_episode in range(1, num_episodes+1):
    state = agent.reset_episode() # start a new episode
    while True:
        action = agent.act(state)
        next_state, reward, done = task.step(action)
        agent.step(reward, done)
        state = next_state
        if done:
            print("\rEpisode = {:4d}, score = {:7.3f} (best = {:7.3f}), noise_scale = {}".format(
                i_episode, agent.score, agent.best_score, agent.noise_scale), end="")  # [debug]
            break
    sys.stdout.flush()
# -

# This agent should perform very poorly on this task.  And that's where you come in!

# ## Define the Task, Design the Agent, and Train Your Agent!
#
# Amend `task.py` to specify a task of your choosing.  If you're unsure what kind of task to specify, you may like to teach your quadcopter to takeoff, hover in place, land softly, or reach a target pose.  
#
# After specifying your task, use the sample agent in `agents/policy_search.py` as a template to define your own agent in `agents/agent.py`.  You can borrow whatever you need from the sample agent, including ideas on how you might modularize your code (using helper methods like `act()`, `learn()`, `reset_episode()`, etc.).
#
# Note that it is **highly unlikely** that the first agent and task that you specify will learn well.  You will likely have to tweak various hyperparameters and the reward function for your task until you arrive at reasonably good behavior.
#
# As you develop your agent, it's important to keep an eye on how it's performing. Use the code above as inspiration to build in a mechanism to log/save the total rewards obtained in each episode to file.  If the episode rewards are gradually increasing, this is an indication that your agent is learning.
#
# ---
#
# I want to implement TD3. Pseudocode is:
#
# ![TD3 Algorithm](td3-pseudocode.png)
#
# Reference Implementation is on [GitHub](https://github.com/sfujim/TD3).
#
# ---

# +
## TODO: Train your agent here.

# needed imports -> I decided to use PyTorch to learn something not seen in the nanodegree until now
import torch.nn.init as init
from agents import TD3Agent
from tasks import Task
from utils import ReplayBuffer
from collections import deque

# set global configuration DICT
config = {
    'INIT_FN_ACTOR': init.uniform_,       # function to use for weight initialization, see torch.nn.init
    'INIT_W_MAX_ACTOR': 0.003,            # maximum value to use if uniform initialization of actor is used
    'INIT_W_MIN_ACTOR': -0.003,           # minimum value to use if uniform initialization of actor is used
    'INIT_W_ACTOR': 0.,                   # fixed value to use if init.constant_ initialization is used
    'INIT_FN_CRITIC': init.uniform_,      # function to use for weight initialization, see torch.nn.init
    'INIT_W_MAX_CRITIC': 0.003,           # maximum value to use if uniform initialization of critic is used
    'INIT_W_MIN_CRITIC': -0.003,          # minimum value to use if uniform initialization of critic is used
    'INIT_W_CRITIC': 0.,                  # fixed value to use if init.constant_ initialization is used
    'LR_ACTOR': 0.0001,                   # learning rate of actor optimizer
    'LR_CRITIC': 0.001,                   # learning rate of critic optimizer
    'WEIGHT_DECAY': 0.0005,               # weight decay of both optimizers
    'BATCH_SIZE': 100,                    # size of mini-batch to fetch from memory store during training
    'DISCOUNT': 0.99,                     # discount factor (gamma) to use
    'TAU': 0.001,                         # factor to use for soft update of target parameters
    'POLICY_NOISE': 0.2,                  # noise added to policy smoothing (sections 5.3, 6.1)
    'NOISE_CLIP': 0.5,                    # noise clip for policy smoothing (sections 5.3, 6.1)
    'REWARD_SCALE': 1.0,                  # reward scaling factor used
    'POLICY_FREQ': 2,                     # update frequence for target networks (sections 5.2, 6.1)
    'ACTION_REPEAT': 3,                   # how often the Task should get next_timestamp per step
    'BUFFER_SIZE': 1_000_000,             # replay memory buffer size
    'EXPLORATION_NOISE': 0.5,             # noise from normal distribution to add during exploration (paper table 3)
    'NUM_EPISODES': 2000,                 # total number of episodes
    'TASK_RUNTIME': 5.0,                  # the time horizon of a single task
}

# create global list to hold episodic rewards -> for logging
all_episode_reward = list()

# create global deque to hold just the last 100 rewards -> for logging
last_rewards = deque(maxlen=100)

# initialize the task to solve
# the quadcopter should forward in every direction
task = Task(init_pose=np.array([10., 10., 0., 0., 0., 0.]),
            init_velocities=np.array([0.1, 0.1, 0.]),
            init_angle_velocities=np.array([0., 0., 0.]),
            target_pos=np.array([20., 20., 110., 0., 0., 0.]),
            runtime=config.get('TASK_RUNTIME', 5.0),
            action_repeat=config.get('ACTION_REPEAT', 3))

# initialize the agent
agent = TD3Agent(task=task, parameters=config)

# PSEUDO CODE: initialize replay buffer
memory = ReplayBuffer(config.get('BUFFER_SIZE', 1_000_000))
# -

## start training
for i_episode in range(1, config.get('NUM_EPISODES', 1000)+1):
    steps = 0
    episode_reward = 0.
    reward = 0.
    state = agent.reset()
    while True:
        # PSEUDO CODE: select action with exploration noise
        # Paper section 6.1 states that a Gaussian noise instead
        # of Ornstein-Uhlenbeck was used because the latter offered
        # no performance benefits
        noise = np.random.normal(0, config.get('EXPLORATION_NOISE', 0.1), 
                                 size=task.action_size).clip(task.action_low, task.action_high)
        action = agent.act(state)
        action += noise
        
        # PSEUDO CODE: execute action in environment
        next_state, reward, done = task.step(action)
        
        # PSEUDO CODE: store transition tuple (state, action, reward, next_state) in memory buffer
        # ignore 'done' signal if maximum runtime is reached
        done_bool = 0. if task.sim.time + task.sim.dt > task.sim.runtime else float(done)
        memory.add((state, action, reward, next_state, done_bool))
        
        # sum up reward
        episode_reward += reward
        
        # progress state
        state = next_state
        
        # increment step#
        steps += 1
        
        # PSEUDO CODE: After each time step, the networks are trained with a
        # mini-batch of a 100 transitions, sampled uniformly from a replay buffer
        # containing the entire history of the agent.
        # --- like the implementation on GitHub training is delayed until end
        #     of a episode
        if done:
            # add cumulative reward to global lists
            all_episode_reward.append(episode_reward)
            last_rewards.append(episode_reward)
            
            # train using a mini batch as often as steps done in this episode
            agent.update(memory=memory, episode_steps=steps)
            
            # print progress after each episode
            print('\rEpisode {:4d} used {:4d} steps, reward = {:7.3f}, average (100) = {:7.3f}, [{:3d}][{:3d}][{:3d}]'.format(
                i_episode, steps, episode_reward, np.mean(last_rewards), int(task.sim.pose[0]), int(task.sim.pose[1]), int(task.sim.pose[2])), end="")
            break
    sys.stdout.flush()

# ## Plot the Rewards
#
# Once you are satisfied with your performance, plot the episode rewards, either from a single run, or averaged over multiple runs. 

## TODO: Plot the rewards.
import seaborn as sns
sns.set()
plt.figure(figsize=(15,5))
plt.grid(True)
plt.plot(all_episode_reward, '.', alpha=0.5, color='red')
plt.plot(np.convolve(all_episode_reward, np.ones(21)/21)[(21-1)//2:-21], color='red', label='Reward per Episode')
plt.ylabel('Reward')
plt.xlabel("Episode #")
plt.legend(loc=2)
plt.xlim(0, len(all_episode_reward))
plt.show()

# ## Reflections
#
# **Question 1**: Describe the task that you specified in `task.py`.  How did you design the reward function?
#
# **Answer**:
#
# I decided to let the Quadcopter start at an initial position and take some units in every direction, the largest in the Z-direction. 
#
# In the first versions rewards were given in a _positive_ way. The idea was to only punish the agent if the quadcopter hits the wall. That means if the position (either x,y, or z) is at the lower or upper bounds of the environment. The reward was designed as 
#
# - start with a zero reward
# - calculate the distance between the points (current position and target position) and add this distance between the points as reward if the new distance is smaller than the initial distance. For example: if the distance between target and initial position is 10 and the distance between target and current position is 2 then the reward is 8. This reward is only given if the distance between the quadcopter and the target position decreased. In addition this reward is multiplied by 100 to get a high reward if the distance decreases.
# - the only punishment is to subtract 10 points if the wall is touched
# - for every x, y, or z position that matches the target position a reward of 1 is added
# - for positive velocity in any direction a reward of 1 is added
# - if x, y, and z positions of the target position are reached a reward of 1000 is given
#
# Additionally I tried to punish the distance by subtract the calculated distance of the reward if the distance increased. Hence the code was
#
#         if current_distance < initial_distance:
#             reward += initial_distance - current_distance
#         else:
#             reward -= current_distance
#
# But adding the `else` branch did not improve the performance.
#
# --- 
#
# Finally I decided to give no reward for constant running and only add some small rewards (1) whenever one of the positions are reached (x, y, or z). If all positions were reached at the same time I add 10 to the reward.

# **Question 2**: Discuss your agent briefly, using the following questions as a guide:
#
# - What learning algorithm(s) did you try? What worked best for you?
# - What was your final choice of hyperparameters (such as $\alpha$, $\gamma$, $\epsilon$, etc.)?
# - What neural network architecture did you use (if any)? Specify layers, sizes, activation functions, etc.
#
# **Answer**:
#
# As written above I decided to implement `Twin Delayed Deep Deterministic Policy Gradients (TD3)`. This decision was made by just reading a lot about reinforcement learning. I first read 
#
# - [Reinforcement learning](https://en.wikipedia.org/wiki/Reinforcement_learning)
# - [Policy Gradient Algorithms](https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html)
#
# After that I read about the mentioned algorithms. I did not read all papers in detail, but skimmed all of them to find the submitted solution.
#
# - [Continuous control with deep reinforcement learning](https://arxiv.org/abs/1509.02971v5)
# - [Trust Region Policy Optimization](https://arxiv.org/abs/1502.05477)
# - [Asynchronous Methods for Deep Reinforcement Learning](https://arxiv.org/abs/1602.01783)
# - [Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation](https://arxiv.org/abs/1708.05144)
# - [Continuous Deep Q-Learning with Model-based Acceleration](https://arxiv.org/abs/1603.00748)
# - [OpenAI Baselines:ACKTR & A2C](https://openai.com/blog/baselines-acktr-a2c/)
# - [Off-policy evaluation for MDPs with unknown structure](https://arxiv.org/abs/1502.03255)
# - [Reinforcement Learning (DQN) Tutorial](https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html)
# - [Prioritized Experience Replay](https://arxiv.org/abs/1511.05952)
# - [Hindsight Experience Replay](https://arxiv.org/abs/1707.01495)
# - [Learning Multi-Level Hierarchies with Hindsight](https://arxiv.org/abs/1712.00948)
# - [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347)
# - [Distributed Distributional Deterministic Policy Gradients](https://arxiv.org/abs/1804.08617)
# - [Addressing Function Approximation Error in Actor-Critic Methods](https://arxiv.org/abs/1802.09477)
# - [Learning to reach by reinforcement learning using a receptive field based function approximation approach with continuous actions](https://link.springer.com/article/10.1007/s00422-009-0295-8)
#
# As the **Instructions and Hints** in the classroom suggested _DDPG_ I decided to implement TD3 because this tries to address some of the *issues* found in DDPG. And I have to admit that I didn't want to just copy and paste the code given by Udacity.
#
# Unfortunately I made several mistakes during the implementation (read below) and I am short of time now. Hence I did not try other algorithms. I wanted to compare TD3 with [SAC](https://arxiv.org/abs/1812.05905) and/or [PPO](https://arxiv.org/abs/1707.06347) but only implemented TD3.
#
# The agent is using PyTorch as I read that PyTorch and Caffe2 were joined. And as we only used Keras and Tensorflow I wanted to learn a bit of PyTorch as well.
#
# Pseudo code of TD3 is given above. I just followed this for the implementation. The reference implementation used only one critic network logically split into two. That means that the first three layers are used as Q1 and the other three as Q2. I implemented both critics as separate models.
#
# The replay buffer used first was a simple deque based one. Initially the neural nets used were using just the same layout as in the research paper of TD3.
#
# Actor networks:
#
#     (state dim, 400)
#     ReLU
#     (400, 300)
#     ReLU
#     (300, action dim)
#     tanh
#     
# Critic networks:
#
#     (state dim + action dim, 400)
#     ReLU
#     (400, 300)
#     RelU
#     (300, 1)
#
# In the run above I switched the layout as inspired by [CEM-RL](https://arxiv.org/abs/1810.01222) to
#
# Actor networks
#
#     (state dim, 400)
#     tanh
#     (400, 300)
#     tanh
#     (300, action dim)
#     tanh
#     
# Critic networks
#
#     (state dim + action dim, 400)
#     Leaky ReLU
#     (400, 300)
#     Leaky RelU
#     (300, 1)
#
# Additionally I implemented a ReplayBuffer that uses a softmax-Function over the rewards saved in the buffer. That way it samples the transitions having the highest rewards first. 
#
# The hyper parameters used first are the same parameters that were used in the research paper as well. 
#
# | Parameter | value   |
# |-----------|---------|
# | $\gamma$ | 0.99 |
# | $\tau$ | 0.05 |
# | $\epsilon$ | $\mathcal{N}$ (0, 0.2) clipped to (-0.5, 0.5) |
# | $\alpha$ of actor optimizer  | 0.01 |
# | $\alpha$ of critic optimizer | 0.01 |
# | Optimizer of all networks | Adam |
# | Reward scaling | 1.0 |
# | Exploration noise | $\mathcal{N}$ (0, 0.1) |
# | Mini-Batch size | 100 |
#
# As written in the research paper I used a update frequency of **2**, hence the target networks are only updated every second learning cycle.
#
# Additionally I decided to manually initialize the network weights randomly drawn from a uniform distribution from (-0.005, 0.005) without weight decay for the optimizer.
#
# During training the agent had some problems (read below) and the hyper parameters finally used can be seen in the config dict above.

# **Question 3**: Using the episode rewards plot, discuss how the agent learned over time.
#
# - Was it an easy task to learn or hard?
# - Was there a gradual learning curve, or an aha moment?
# - How good was the final performance of the agent? (e.g. mean rewards over the last 10 episodes)
#
# **Answer**:
#
# As seen in the reward plot and the final position of the quadcopter the task was not learned by this agent. I think that I choosed one of the hardest tasks: Starting at one point and blindly find another one. I assume that lifting or landing the quadcopter is easier to train. 
#
# I assume that I did not find a reward function good enough to help the agent. The other things I tried: 
#
# - running for more number of periods (2000 instead of 1000)
# - changed the network architecture
# - changed learning rates of actor and critics
# - changed discount
# - changed tau
# - changed exploration noise
# - changed replay buffer implementation
# - changed the task runtime

# **Question 4**: Briefly summarize your experience working on this project. You can use the following prompts for ideas.
#
# - What was the hardest part of the project? (e.g. getting started, plotting, specifying the task, etc.)
# - Did you find anything interesting in how the quadcopter or your agent behaved?
#
# **Answer**:
#
# This project was very hard for me. I realized that I did not fully understand the concepts during the classroom sessions and the mini projects. I finally get the concepts after reading a lot of papers (not only the list above) and blog posts (mainly on medium.com). 
#
# Then I tried to start by implementing some of the replay buffer concepts first as this class is used as utility class and I wanted to have that done first. I tried the simple replay buffer (easy to understand), the rank based prioritized replay buffer (I used a heapq), the td error based prioritized replay buffer (using a binary segment tree) and implemented two by myself that used the gained rewards as priorization. The first one works as:
#
#     split the length of the deque into sample_size / 2 segments
#     for every segment
#         get the index of the sample that contains the highest reward
#         add this sample to the return list
#         if this index is at the first position of the segment
#             add the next sample to the return list
#         else
#             add the previous sample to the return list
#     return sample list
#
# The second one just stores the observations in two deque objects: One holding all observations and the second holding only the rewards gained. During sampling I use `softmax` to squeeze the rewards into the probability range 0-1 and use that with `numpy.random.choice` to sample the observations having the highest rewards (what means that training using the sample are biased because most of the time the same observations will be sampled over and over again.)
#
# After that I needed to learn how to use PyTorch. Thanks to the excellent documentation this was easy enough. I implemented the TD3 agent and the Actor and Critic models as described above. But something went wrong. I made two errors:
#
# 1. the usage of the replay buffer was not implemented correct
# 2. I did some errors implementing and training the Critic networks
#
# Fixing that took too much time. Hence I just come back to use the softmax replay buffer (that uses the same structure as the simple replay buffer) and finally got the Critic correct (at least I think so). 
#
# One very bad problem remaines: Whatever I do my agent stucks. As written above I changed the reward function several times. I changed the hyper parameters several times. I change the network architecture. But the agent hucks the quadcopter to the area boundary very soon somtimes and then learns to stay with that. For example in several runs the Z-Position was trained to 300. Or the Y-Position remained at 0 all the time. I change the Z-position for inital and target position and set the initial velocity in the Z-Direction to 0. That just freezes the Z-position to around 127 (instead of 110). X and Y positions are not learned at all. 
#
# I assume that I'm not able to find a reward function that helps the agent to understand the task at hand. But I do not want to train a startup-task or a hover task. It should be possible to train the agent to find the target position.
#
# In addition I did not understand `physics_sim.py`. Perhaps I need to change state or action size. But I have no idea about how to do that. 

StarcoderdataJupyterScriptsDedupFiltered