abstract:  Model-based Reinforcement Learning (MBRL) allows data-efficient learning which is required in real world applications such as robotics. However, despite the impressive data-efficiency, MBRL does not achieve the final performance of state-of-the-art Model-free Reinforcement Learning (MFRL) methods. We leverage the strengths of both realms and propose an approach that obtains high performance with a small amount of data. In particular, we combine MFRL and Model Predictive Control (MPC). While MFRL's strength in exploration allows us to train a better forward dynamics model for MPC, MPC improves the performance of the MFRL policy by sampling-based planning. The experimental results in standard continuous control benchmarks show that our approach can achieve MFRL's level of performance while being as data-efficient as MBRL.
author: Zhang-Wei Hong
National Tsing Hua University
`firstname.lastname@example.com` Joni Pajarinen
TU Darmstadt
`email@example.com` Jan Peters
TU Darmstadt and Max Planck Institute for Intelligent Systems
`email@example.com`
bibliography: neurips_2019.bib
title: Model-based Lookahead Reinforcement Learning

# Introduction

Model-free Reinforcement Learning (MFRL) has succeeded in several domains, including video game playing  and robot control . However, high sample complexity prevents applying MFRL to most complex real world applications: MFRL directly optimizes the agent's policy from interactions with the environment usually requiring millions of data samples .

Contrary to MFRL, model-based reinforcement learning (MBRL) is typically more data-efficient. However, when the perfect model is inaccessible, a forward dynamics model of the environment must be approximated . Despite the learning of the forward dynamics model, MBRL often requires less interactions with the environment compared to learning a policy directly. Nevertheless, a major disadvantage of MBRL is that, due to model approximation errors , MBRL commonly cannot achieve the performance of MFRL at convergence.

Prior work takes advantage of both MFRL and MBRL to some extent. show that generating training samples for MFRL by model-based trajectory optimization may increase the performance of MFRL, while  find in contrast that insufficient exploration of MBRL may impair the performance of the resultant policy and   show that naive exploration for training forward dynamics models prevents accuracy of the acquired forward dynamics model. Apart from generating samples using MBRL, combine MFRL with online model-based planning (i.e. MBRL) to improve the performance of the reinforcement learning (RL) agent, but either is restricted in discrete state and action spaces, relies on the assumption that the state space has 2-D structure, or assumes a perfect forward dynamics model. In sum, despite the notable performance of the conjunction of MFRL with model-based planning, the strong assumptions prohibit applications on more complex tasks.

We propose a novel framework that unifies MFRL and MBRL through Model Predictive Control (MPC). Our approach leverages the merits of MFRL and MBRL at both training and testing time. At training time, we utilize the exploratory policy of MFRL to collect more diverse training samples in the environment than MBRL  and thus prevent the impacts of insufficient exploration on policies and forward dynamics models. Next, at testing time, we combine sampling-based MPC (a model-based online planning approach) with MFRL to further increase the performance beyond training. Different from the contemporary works , our approach does not rely on any assumption of perfect forward dynamics models or state/action space. We use an approximated forward dynamics model that can be applied in arbitrary state/action spaces. Furthermore, we jointly leverage the value function and the policy to yield remarkable MPC planning performance and data-efficiency. Finally, we propose a soft-greedy approach that improves action selection in planning. To our best knowledge, we are the first to combine MFRL and MPC, on the least assumptions of state/action space.

We demonstrate the effectiveness of our approach on well-known challenging continuous control tasks in MuJoCo . The experimental results show that our approach leads to better performance in less data than that of independently using MFRL or MPC, particularly in complex tasks, and thus confirm that our approach of combining MBRL and MFRL is effective. Furthermore, the results show that our approach favors the quality of forward dynamics model training. In addition, we provide empirical analysis to verify the limitations of contemporary approaches.

The contributions of this paper are the following:

-   We show that using an MFRL policy to collect training data improves the quality of the trained forward dynamics model.

-   We show that using an MFRL policy can enhance MPC' s performance.

-   We show advantages and limitations of using value function in MPC.

-   We provide a comprehensive evaluation of each design decision for combining MPC and MFRL.

The remaining parts of this paper are organized as follows. Section  introduces the background. Section  describes our new approach. Section  presents and analyzes the experimental results. Finally, Section  concludes the paper.

# Background

In order to elaborate the motivation and the implementation of the proposed method, this section starts with an introduction to Reinforcement Learning (RL) , then explains MFRL, and finally discusses recently proposed MPC approaches for MBRL.

## Reinforcement Learning (RL)

In this paper, we study a standard deterministic discrete time RL problem, consisting of a 4-tuple $(\mathcal{S}, \mathcal{A}, R, f)$. $\mathcal{S}$ denotes the state and $\mathcal{A}$ the action space. $R:\mathcal{S} \times \mathcal{A} \mapsto \mathbb{R}$ is a task-specific reward function encoding the task objective. $f:\mathcal{S} \times \mathcal{A} \mapsto \mathcal{S}$ is the true forward dynamics model. At each time step $t$, an RL agent perceives the current state $\boldsymbol{s}_t \in \mathcal{S}$, takes the control action $\boldsymbol{a}_t \in \mathcal{A}$ and observes the next state $\boldsymbol{s}_{t+1} = f(\boldsymbol{s}_t, \boldsymbol{a}_t)$ and the immediate reward $R(\boldsymbol{s_t}, \boldsymbol{a}_t)$. The training objective of RL is to search for an optimal controller that selects $\boldsymbol{a}_t$ for each $\boldsymbol{s}_t$ such that the expected return $G_t = \sum_{i=t}^{T} \gamma^{i-t} R(\boldsymbol{s}_i, \boldsymbol{a}_i)$ is maximized, where $T$ represents the horizon of the task and $\gamma \in (0,1)$ denotes the constant discount factor. Note that while this paper adheres to deterministic cases for simplicity our framework can be easily extended to a stochastic formulation.

## Model-free Reinforcement Learning (MFRL)

We discuss both the training and evaluation phases in MFRL. First, using data $(\boldsymbol{s}_t, \boldsymbol{a}_t, \boldsymbol{s}_{t+1})$ collected by an exploratory policy $\pi^\prime$, the training phase learns a possibly stochastic control policy $\pi_{\boldsymbol{\theta^\pi}}: \mathcal{S} \times \mathcal{A} \mapsto [0, 1]$ that infers $\boldsymbol{a}_t$ which maximizes $G_t$ for $\boldsymbol{s}_t$, and a value function $V_{\boldsymbol{\theta^V}}: \mathcal{S} \mapsto \mathbb{R}$ estimating expected returns for $\boldsymbol{s_t}$. $\boldsymbol{\theta^\pi}$ and $\boldsymbol{\theta^V}$ denote parameters. In this paper, we adhere to an appealing MFRL approach, policy gradient  which updates $\boldsymbol{\theta^\pi}$ along the direction $\nabla_{\boldsymbol{\theta^\pi}}  \text{log}(\pi_{\boldsymbol{\theta^\pi}}(\boldsymbol{a}_t|\boldsymbol{s}_t)) (G_t - b(\boldsymbol{s}_t))$. $b(\boldsymbol{s}_t)$ is for variance reduction . A common choice for $b(\boldsymbol{s}_t)$ is the value function estimate $V_{\boldsymbol{\theta}^V}(\boldsymbol{s}) \approx \mathbb{E}_{\boldsymbol{s}_{t+1:T}, \boldsymbol{a}_{t:T} \sim \pi_{\boldsymbol{\theta^\pi}}}\big[G_t | \boldsymbol{s_t} = \boldsymbol{s} \big]$. $\boldsymbol{\boldsymbol{\theta}^V}$ can be obtained by various value function approximation approaches . Finally, the evaluation phase simply samples control actions $\boldsymbol{a}_t$ using $\pi_{\boldsymbol{\theta}^\pi}$: $\boldsymbol{a}_t \sim \pi_{\boldsymbol{\theta}^\pi}(\boldsymbol{a}|\boldsymbol{s}_t)$.

## Model Predictive Control for Model-based Reinforcement Learning (MPC-MBRL)

MPC has long been prevalent in robotic control  and recently been applied to MBRL . MPC-MBRL simply trains a forward dynamics model to plan the control action $\boldsymbol{a}_t$ at each time step during evaluation.

The training phase trains an approximated forward dynamics model $f_{\boldsymbol{\theta^f}}$ since the real forward dynamics model $f$ is usually inaccessible . MPC-MBRL collects an initial dataset $\mathcal{D}$ consisting of a series of transitions $(\boldsymbol{s}_t, \boldsymbol{a}_t, \boldsymbol{s}_{t+1})$ by using uniformly random exploration in the environment: $\boldsymbol{a}_t \sim \mathcal{U}(\mathcal{A})$, then optimizing $f_{\boldsymbol{\theta^f}}$ by minimizing the following loss function: $$\label{eq::fwd_dynamic_loss}
    L_{{f_{\boldsymbol{\theta}^f}}}(\boldsymbol{\theta}_f) = \frac{1}{\lVert {B} \rVert} \sum_{\boldsymbol{s}_t, \boldsymbol{a}_t, \boldsymbol{s}_{t+1} \in {B}}{{\lVert \boldsymbol{s}_{t+1} - {f_{\boldsymbol{\theta}^f}}(\boldsymbol{s}_t, \boldsymbol{a}_t) \rVert}^2},$$ where $B$ is a batch of transitions $(\boldsymbol{s}_t, \boldsymbol{a}_t, \boldsymbol{s}_{t+1})$ sampled from $\mathcal{D}$. Next, MPC-MBRL appends data from online execution of MPC-planning to $\mathcal{D}$: $\boldsymbol{a}_t = \text{MPC}(\boldsymbol{s}_t)$ , and trains the forward dynamics model $f_{\boldsymbol{\theta^f}}$ according to Eq.  again.

In the evaluation phase, MPC-MBRL computes at each time step the control action $\boldsymbol{a}_t$ for the current state $\boldsymbol{s}_t$, with model-based planning consisting of three stages: trajectory sampling, trajectory evaluation, and action selection. Trajectory sampling stage simulates a set of trajectories: $\Phi = \{\hat{\tau}^n = [\boldsymbol{\hat{s}}^n_1, \boldsymbol{\hat{a}}^n_1, \boldsymbol{\hat{s}}^n_2, \cdots, \boldsymbol{\hat{s}}^n_{H+1}]\}^{N}_{n=1}$, where $N$ denotes the number of simulated trajectories, $n$ denotes the index of a trajectory within $\Phi$, and $H$ indicates the planning horizon. $\tau^n$ can be obtained by sequentially applying $$\begin{aligned}
    \label{eq::traj_sampling}
        \boldsymbol{\hat{s}}^n_1 &= \boldsymbol{s}_t; ~  \boldsymbol{\hat{s}}^n_{h+1} = f_{\boldsymbol{\theta^f}}(\boldsymbol{\hat{s}}^n_h, \boldsymbol{\hat{a}}^n_h); ~ \boldsymbol{\hat{a}}^n_h \sim \mathcal{Z},
    \end{aligned}$$ where $\boldsymbol{\hat{s}}^n_h$ denotes the simulated state at planning step $h$ within $\hat{\tau}^n$, $\boldsymbol{\hat{a}}^n_h$ is the action applied to $\boldsymbol{\hat{s}}^n_h$, and $\mathcal{Z}$ denotes a given action distribution. Next, trajectory evaluation stage evaluates each trajectory $\hat{\tau}^n$ with the task-specific reward function $R$ and the terminal reward function $R_{\phi}$: $$\label{eq::traj_eval}
     \hat{G}(\boldsymbol{\hat{s}}^n_{1:H+1}, \boldsymbol{\hat{a}}^n_{1:H}) =  {\sum^{H}_{h = 1}{\gamma^{h - 1}R(\boldsymbol{\hat{s}}^n_h, \boldsymbol{\hat{a}}^n_h})} + \gamma^{H} R_{\phi}(\boldsymbol{\hat{s}}^n_{H}),$$ where $\hat{G}(\boldsymbol{\hat{s}}^n_{1:H+1}, \boldsymbol{\hat{a}}^n_{1:H})$ denotes the simulated accumulated rewards associated with $\hat{\tau}^n$, and $\boldsymbol{\hat{s}}^n_{1:H+1}$ and $\boldsymbol{\hat{a}}^n_{1:H}$ denote the state and action sequences extracted from $\hat{\tau}^n$ respectively. Although $R_{\phi}$ is typically ignored (i.e. $R_{\phi}(.) = 0$), we use $R_{\phi}$ to illustrate our approach in Section . Finally, action selection stage selects the first control action $\boldsymbol{a}^*_t$ of the action sequence $\boldsymbol{a}^*_{t:t+H}$ which yielded highest value w.r.t. Eq. : $$\label{eq::act_select}
    \boldsymbol{a}^*_{t:t+H} =  \mathop{\mathrm{argmax}}_{\boldsymbol{\hat{a}}^n_{1:H}}{{\hat{G}(\boldsymbol{\hat{s}}^n_{1:H+1}, \boldsymbol{\hat{a}}^n_{1:H})}}.$$

# The approach: Model Predictive Control with Model-free Reinforcement Learning (MPC-MFRL)

Typical MFRL algorithms and MBRL methods do not fully utilize the beneficial information that can be extracted from the environment. MFRL ignores the information encapsulated in the forward dynamics of the environment, while MPC neglects the utility of policies and value functions. Contrary to both, our approach jointly leverages forward dynamics, policies, and value functions, and therefore is more likely to have better performance in the case of dearth of data. Section  details the proposed training method and Section  the proposed online hybrid MPC-MFRL approach.

## The training phase: learning a policy, value function, and a forward dynamics model

We jointly train the control policy $\pi_{\boldsymbol{\theta}^\pi}$, the value function $V_{\boldsymbol{\theta}^V}$, and the forward dynamics model $f_{\boldsymbol{\theta}^{f}}$ using the same data at the same time. The training phase iteratively executes the following three steps until convergence. Firstly, we gather a trajectory $\tau_{\pi^\prime} = [\boldsymbol{s}_1, \boldsymbol{a}_1, \boldsymbol{s}_{2}, \cdots, \boldsymbol{s}_{T+1}]$ using the exploratory policy $\pi^\prime$ (if using on-policy RL, $\pi^\prime$ must be set as ${\boldsymbol{\theta}^\pi}$). $\pi^\prime$ collects data by interacting with the environment: $\boldsymbol{a}_t \sim \pi^\prime(\boldsymbol{a}|\boldsymbol{s_t})$ (as Section  describes). Secondly, we update ${\boldsymbol{\theta}^\pi}$ and ${\boldsymbol{\theta}^V}$ by $\tau_{\pi^\prime}$, as Section  describes. In this paper, we use Trust Region Policy Optimization (TRPO) , a trust region policy gradient method, for training $\pi_{\theta^\pi}$ since TRPO has been successful in several domains and is a theoretic sounded policy gradient algorithm, but other MFRL methods  could be used instead. Finally, we append $\tau_{\pi^\prime}$ to the training dataset $\mathcal{D}$: $\mathcal{D} \leftarrow \mathcal{D} \cup \{(\boldsymbol{s}_t, \boldsymbol{a}_t, \boldsymbol{s}_{t+1})  \}^T_{t=1}$ ($\mathcal{D}$ is initialized as $\emptyset$), and optimizes the forward dynamics model $f_{\boldsymbol{\theta}_f}$ by minimizing the loss function defined in Eq.  with batches $B$ sampled in $\mathcal{D}$. Algorithm 5 in supplementary material details the training scheme. Note that though this paper focuses on training a deterministic forward dynamics model, probabilistic models  or non-parametric models  can be easily applied in our approach.

Jointly training $\pi_{\boldsymbol{\theta}^\pi}$, $V_{\boldsymbol{\theta}^V}$, and $f_{\boldsymbol{\theta}^f}$ poses the following advantages. First and foremost, $\pi^\prime$ can collect more extensive data for the forward dynamics model than does uniform random exploration and MPC on-policy data aggregation . Optimized to maximize reward, $\pi_{\boldsymbol{\theta}^\pi}$ can collect successful experiences which uniform random exploration cannot. Also, MPC simply exploits rewards while an MFRL exploratory policy balances exploitation and exploration and thereby guarantees data diversity. Second, interacting with an environment using $\pi^\prime$ requires less computation time than collecting the training dataset using MPC-planning . Finally, our joint training procedure maximize the utility of each interaction with the environment, comparing to simply learning a policy (MFRL) or a forward dynamics model (MBRL).

## The evaluation phase: planning the control action using the policy, the value function, and the forward dynamics model

Our approach at the evaluation time is built on the top of MPC framework defined in Section  and improves each stage by MFRL. We use MFRL's control policy for trajectory sampling, MFRL's value function for trajectory evaluation, and soft-greedy approach for action selection. Fig.  illustrates the planning process and Algorithm  summarizes the approach. Next, we discuss how our method improves trajectory sampling, trajectory evaluation, and action selection.

#### Trajectory sampling.

  By replacing the action distribution $\mathcal{Z}$ in Eq.  with the MFRL control policy $\pi_{\boldsymbol{\theta}^\pi}$, our method can more efficiently sample trajectories of high value than uniform random sampling  and Cross Entropy Method (CEM) . We sample as follows: $$\begin{aligned}
    \label{eq::new_traj_sampling}
        \boldsymbol{\hat{s}}^n_1 &= \boldsymbol{s}_t; ~ \boldsymbol{\hat{s}}^n_{h+1} = f_{\boldsymbol{\theta}^f}(\boldsymbol{\hat{s}}^n_h, \boldsymbol{\hat{a}}^n_h); ~ \boldsymbol{\hat{a}}^n_h \sim \pi_{\boldsymbol{\theta}^\pi}(\boldsymbol{a}|\boldsymbol{\hat{s}}^n_h).
    \end{aligned}$$ The MFRL control policy $\pi_{\boldsymbol{\theta}^\pi}$ improves trajectory sampling in the following aspects. To begin with, $\pi_{\boldsymbol{\theta}^\pi}$ can readily result in high-value trajectories since $\pi_{\boldsymbol{\theta}^\pi}$ is trained to maximize the expected return $G_t$. Additionally, even before convergence $\pi_{\boldsymbol{\theta}^\pi}$ can restrain the search space in trajectory sampling and thus allows trajectories of high value to be sampled more likely than in uniform sampling . Moreover, in contrast to CEM  which uses simulated data for optimization, our method is not susceptible to forward dynamics approximation error since $\pi_{\boldsymbol{\theta}^\pi}$ is trained with real experience. Finally, our method does not require costly computations since our method does not rely on online iterative optimization like CEM .

#### Trajectory evaluation.

  Similar to  , our approach substitutes the terminal reward function $R_{\phi}$ (Eq. ) with the MFRL value function $V_{\boldsymbol{\theta}^V}$ to resolve the shortsighted planning problem mentioned in Section . Trajectory evaluation becomes: $$\label{eq::new_traj_eval}
     \hat{G}(\boldsymbol{\hat{s}}^n_{1:H+1}, \boldsymbol{\hat{a}}^n_{1:H}) =  {\sum^{H}_{h = 1}{\gamma^{h - 1}R(\boldsymbol{\hat{s}}^n_h, \boldsymbol{\hat{a}}^n_h)}} + \gamma^{H} V_{\boldsymbol{\theta}^V}(\boldsymbol{\hat{s}}^n_{H}).$$ The MFRL value function $V_{\boldsymbol{\theta}^V}$ in Eq. estimates the expected return of a given state (see Section ) and therefore prevents shortsighted planning even with a short planning horizon $H$. More importantly, planning with a short horizon avoids compounding errors in simulation of long horizon and saves computation time as well.

Different from  , we use an approximated dynamics model rather than a perfect one. The assumption of a perfect dynamics model is unrealistic in most cases, especially on complex tasks, where the perfect dynamics model is inaccessible. In addition, estimating the expected return of a simulated state could be risky since the value function trained with real experience is unlikely to be accurate in the states absent in the training data. Moreover, jointly using value function estimation and a plain random action sampling in simulation of MPC-planning could magnify this problem since those unconstrained or weakly constrained action sampling may induce lots of unreachable states (e.g. states that violate physical constraints) where the value function cannot estimate accurately on.

However, our approach can alleviate the above problem by sampling actions with an MFRL policy in trajectory sampling. MFRL policies can prevent agents from performing actions that could lead to unreachable states since MFRL polices are trained to maximize to expected reward of the task and thus are less likely to perform those infeasible actions.

#### Action selection.

  If we simply take the control action as the sampled action that yielded the max expected return (Eq. ), the agent may be overly optimistic to the simulated results and is likely to impair the performance. Thus, we propose a soft-greedy approach to alleviate the impact of approximation errors in the forward dynamics model $f_{\boldsymbol{\theta}^f}$. Our soft-greedy approach takes the control action $\boldsymbol{a}_t$ as the average over the $E$ best action sequences w.r.t. $\hat{G}(\boldsymbol{\hat{s}}^n_{1:H+1}, \boldsymbol{\hat{a}}^n_{1:H})$. Formally, we describes as the follows: $$\label{eq::new_act_select}
    \begin{split}
            \bar{\boldsymbol{a}}^{1:N}_{1:H} &=  \mathop{\mathrm{argsort}}_{\boldsymbol{\hat{a}}^n_{1:H}}{{\hat{G}(\boldsymbol{\hat{s}}^n_{1:H+1}, \boldsymbol{\hat{a}}^n_{1:H})}}. \\
            \boldsymbol{a}^*_{t:t+H} &= \frac{1}{E} \sum_{e=1}^{E}\bar{\boldsymbol{a}}^{e}_{1:H}, \\
    \end{split}$$ where $\mathop{\mathrm{argsort}}$ sorts all action sequences $\boldsymbol{\hat{a}}^{1:N}_{1:H}$ according to $\hat{G}(\boldsymbol{\hat{s}}^n_{1:H+1}, \boldsymbol{\hat{a}}^n_{1:H})$ in descending order.

Our proposed soft-greedy action selection approach can prevent biasing toward the best action obtained from imperfect simulation. Averaging has been shown to be able to alleviate the inherent bias of the max-operator (Eq. ) . Preventing bias due to the max-operator allows MPC-MFRL to operate with inaccurate forward dynamics models caused by, for example, underfitting or overfitting forward dynamics models .

# Experiments

The experiments are designed to answer the main question of whether MPC-MFRL successfully leverages MPC to bridge the gap between MBRL and MFRL, overcoming the drawbacks of prior works. Moreover, we perform additional experiments to answer the following more detailed questions: (1) Do we obtain a more accurate forward dynamics model by collecting training samples using an exploratory policy of MFRL? (2) Does an MFRL control policy enhance the planning performance of MPC? (3) Can the value function favor the performance of MPC planning even in an approximated forward dynamics model? (4) Does the proposed soft-greedy action selection improve performance under forward dynamics model approximation errors? Next, we shortly introduce the experimental setup, and then discuss the experimental results.

## Experimental setup

#### Benchmark tasks.

  We use standard continuous control Mujoco benchmark tasks of varying difficulty from OpenAI gym  varying in dimensions of $\mathcal{S}$ and $\mathcal{A}$: *Swimmer* $(\mathcal{S} \subseteq \mathbb{R}^{10}, \mathcal{A} \subseteq \mathbb{R}^2)$, *Reacher* $(\mathcal{S} \subseteq \mathbb{R}^{11}, \mathcal{A} \subseteq \mathbb{R}^2)$, *HalfCheetah* $(\mathcal{S} \subseteq \mathbb{R}^{18}, \mathcal{A} \subseteq \mathbb{R}^6)$, and *Ant* $(\mathcal{S} \subseteq \mathbb{R}^{30}, \mathcal{A} \subseteq \mathbb{R}^8)$.

#### Implementation of MPC-MFRL.

  The MFRL control policy $\pi_{\boldsymbol{\theta^\pi}}$, the MFRL value function $V_{\boldsymbol{\theta^V}}$, and the approximated forward dynamics model $f_{\boldsymbol{\theta^f}}$ are implemented as neural networks. $\pi_{\boldsymbol{\theta^\pi}}$ is modeled as a multi-variate Gaussian distribution, where $\pi_{\boldsymbol{\theta^\pi}}(\boldsymbol{a}|\boldsymbol{s_t}) =  \mathcal{N}( {\boldsymbol{\mu_{\theta^\pi}}}, {\boldsymbol{\Sigma_{\theta^\pi}}} | \boldsymbol{s_t})$, where ${\boldsymbol{\mu_{\theta^\pi}}}$ and ${\boldsymbol{\Sigma_{\theta^\pi}}}$ are the mean vector and the covariance matrix conditioned on learned parameters $\boldsymbol{\theta^\pi}$ and current state $\boldsymbol{s}_t$. We optimize $\pi_{\boldsymbol{\theta^\pi}}$ by TRPO , while updating $V_{\boldsymbol{\theta^V}}$ and $f_{\boldsymbol{\theta^f}}$ using the Adam optimizer .

#### Baselines.

  For comparison against both MFRL and MPC-MBRL methods we select the following baselines (see supplementary material for details):

-   *MF (S)*: TRPO that uses stochastic actions for evaluation.

-   *MF (D)*: TRPO that uses deterministic actions for evaluation (i.e. $\boldsymbol{\mu_{\theta^\pi}}$).

-   *MPC-Random*: MPC that samples actions from a uniform distribution for trajectory roll-outs (Eq. ) and uniform random exploration with on-policy data aggregation (see Section ) for training the forward dynamics model.

-   *MPC-CEM*: MPC that uses the same model training approach as *MPC-Random*, while using CEM  for trajectory sampling since CEM has been shown to work well with MPC in prior work .

## Evaluation procedure

In order to assess performance and data-efficiency of each method, we evaluate offline  each method periodically w.r.t. number of training samples used. At offline evaluation time, we fix all model parameters and measure the average total reward over 10 episodes. We then report the mean and bootstrapped confidence interval of the best average total reward (denoted "Average return" in figures below) so far over 5 distinct random seeds.

## The results of overall performance

Fig.  shows the performance of each method w.r.t. the number of samples. MPC-MFRL achieves better performance than all baseline methods: MPC-MFRL exceeds the performance of MPC-MBRL while being more data-efficient than MFRL. Moreover, the improvement is particularly significant in the more challenging tasks like *Ant* and *HalfCheetah*. To conclude, this result verifies the effectiveness of MPC-MFRL on common benchmark tasks.

Interestingly, Fig.  shows that *MPC-CEM* loses to *MPC-Random* in *Ant-v2* and *HalfCheetah-v2*. Also, we find that even though *MPC-CEM* obtains the highest expected return in simulation (Eq. ), the expected return in the real environment is surprisingly low. Prior works of MBRL  suggest that training a policy using fictitious data impairs performance. Thus, the poor performance of *MPC-CEM* in complex tasks could be caused by optimizing the policy using simulated data of high approximation errors.

## The results of improved exploration for training

We show that our training approach improves the training quality of forward dynamics models (Section ) by comparing various training schemes w.r.t. testing error of the forward dynamics model and evaluation performance. Fig.  shows the testing error of the forward dynamics models trained by different schemes. Half of the testing set consists of data from *Random+MPC* and half from *Policy* (see Section C in supplementary material for more details). *Policy* reduces the testing error faster and more than *Random+MPC*, which shows that collecting data by an MFRL policy can increase the accuracy of forward dynamics models. In addition to favoring accuracy, Fig.  further shows *MPC-MFRL (Policy)* outperforms *MPC-MFRL (Random+MPC)*, especially in later stages of evaluation. To summarize, these results verify our approach is superior to prior approaches in terms of accuracy of forward dynamics models and evaluation performance, also showing that accuracy of forward dynamics models greatly affects the performance.

## The results of sampling trajectories by an MFRL policy

Fig.  shows *MPC-MFRL ($\mathcal{Z} = \pi$)* outperforms *MPC-MFRL ($\mathcal{Z} = U$)* in all tasks and therefore suggests that the MFRL control policy $\pi_{\boldsymbol{\theta}^\pi}$ can more readily sample trajectories of high value in the real environment than the baselines. Furthermore, *MPC-MFRL ($\mathcal{Z} = \pi$)* still surpasses *MPC-MFRL ($\mathcal{Z} = U$)* even in the early stages where the pure model-free variants of MPC-MFRL (i.e. *MF (S/D)*) perform poorly compared to *MPC-MFRL ($\mathcal{Z} = U$)*. This result confirms that even though the MFRL control policy $\pi_{\boldsymbol{\theta}^\pi}$ has not yet converged, our method can still enhance MPC planning performance, and also show the improvement of data-efficiency against MFRL. Note that we ignore comparison with CEM due to its poor performance in Fig. .

## The strength and the limitations of trajectory evaluation with a value function

We investigate the strength and limitations of our trajectory evaluation approach in this section. Fig.  shows that *MPC-MFRL ($R_{\phi}(s_t) = V_{\theta}(s_t), H = 2,5,20$)* outperforms *MPC-MFRL ($R_{\phi}(s_t) = 0, H = 2,5,20$)* with the same planning horizon and thereby verifies that our approach (Eq. ) is effective. However, on the contrary to the results in the prior work , Fig.  shows that  *MPC-MFRL ($R_{\phi}(s_t) = V_{\theta}(s_t), H = 5$)* approximates *MPC-MFRL ($R_{\phi}(s_t) = V_{\theta}(s_t), H = 20$)*, which suggests that compounding errors in a longer planning horizon may impair the performance of value function in an approximated dynamics model. In addition, we find that in the experiments of Section , the terminal reward (i.e. $V_{\theta}(s_t)$) in simulated trajectories of *MPC-MFRL ($\mathcal{Z} = U$)* are similar than *MPC-MFRL ($\mathcal{Z} = \pi$)*. This observation implies that a uniform random sampling may sample states that lead overstimation of value function in approximated dynamics model. The detail investigation are left as future works.

## The results of soft-greedy action selection

This section verifies that our soft-greedy action selection approach improves performance under an approximate forward dynamics model, followed by studying the effectiveness with models of varying complexity. Fig.  shows that *MPC-MFRL (w SG)* outperforms *MPC-MFRL (w/o SG)*, suggesting that soft-greedy action selection increases performance of an approximated forward dynamics model. Fig.  shows that *MPC-MFRL (w SG)* surpasses *MPC-MFRL (w/o SG)* in all model complexities, thereby suggesting that our soft-greedy action selection is superior to classical greedy action selection in both simple and complex models.

# Related work

The classic Dyna framework  learn a forward dynamics model for simulating experiences to train an MFRL agent, while we focus on the interplay of MFRL and MPC.

Guided Policy Search (GPS)  uses model-based controllers such as iLQR  and iLQG  to generate training samples for MFRL policy. The follow-up works  further adopt MPC to provide supervision for an policy. In contrast, our approach concentrates on improving MPC performance with MFRL.

, , and assist value function learning by model-based approaches. collect training samples for value function learning using a model-based controller. and leverage model-based rollout to compute targets of value function training, thus accelerating value function learning. On the contrary, our method focuses on leveraging value function to ameliorate the shortsighted planning of MPC.

, , , and add planning to an MFRL policy. and , however, either rely on discrete state and action spaces or a perfect forward dynamics model. assume that the state space has 2-D structure. learn planning by an MFRL approach, hence inheriting the data-inefficiency of MFRL. In contrast, our approach can be applied to an arbitrary type of state and action space and is more data-efficient than MFRL.

perform MPC planning solely with a goal-conditioned state-action value function. improve long-term planning by evaluating sampled trajectories using a value function. and do not utilize MFRL policies to enhance planning performance. Planning with only a value function  inherits the data-inefficiency of value function learning in MFRL, while our approach combines simulated rewards and a value function, thereby mitigating this problem. assume a perfect forward dynamics model, whereas our work does not rely on such an unrealistic assumption.

# Conclusion

We propose the MPC-MFRL framework which leverages the advantages of MFRL and MPC to achieve MFRL's level of performance while being as data-efficient as MBRL. Moreover, MPC-MFRL allows the agent to continually improve performance with more environment interactions. On the other hand, our novel MPC-MFRL framework brings promising future work to light as well. An application of MPC-MFRL on real robotics systems, for example, can be an appealing direction since MPC-MFRL shows superior data-efficiency particularly crucial for real robots. Another direction could be guiding an MFRL policy by MPC. One can pre-train an MFRL policy in several tasks, then online adapt to a new task by MPC or model-based policy search.


abstract: We have simulated the response of a high energy neutrino telescope in deep Antarctic ice to the stream of low energy neutrinos produced by a supernova. The passage of a large flux of MeV-energy neutrinos during a period of seconds will be detected as an excess of single counting rates in all individual optical modules. We update here a previous estimate of the performance of such an instrument taking into account the recent discovery of absorption lengths of several hundred meters for near-UV photons in natural deep ice. The existing AMANDA detector can, even by the most conservative estimates, act as a galactic supernova watch.
address:  $^1$Department of Physics, University of Wisconsin, Madison, WI 53706, USA
  $^2$Dpto. de Partı́culas, Universidad de Santiago, E-15706 Santiago, Spain
author: F. Halzen$^1$, J.E. Jacobsen$^1$ and E. Zas$^2$
title:  Ultra-Transparent Antarctic Ice as a Supernova Detector

# Introduction

Although aspects of the observations of SN1987A left some lingering doubts about supernova models, they provided, in general, remarkable confirmation of established ideas about supernova mechanisms. At collapse, the core of the progenitor star is expected to release energy in a prompt $\nu_e$ burst lasting a few milliseconds. Most of the energy is, however, liberated after deleptonization in a burst lasting about ten seconds. Roughly equal energies are carried by each neutrino species with a thermal spectrum of temperature $2-4~$MeV. The time scale corresponds to the thermalization of the "neutrinosphere" and its diffusion within the dense core.

Since the ${\bar{\nu}}_e$ cross-section on protons in the detector is significantly larger than the interaction cross sections for the other neutrino flavors, ${\bar{\nu}}_e$ events dominate the signal by a large factor after detection efficiency is taken into account. In this reaction, free protons absorb the antineutrino to produce a neutron and a positron which is approximately isotropically emitted with an energy close to that of the initial neutrino. A thermal spectrum of temperature $4~$MeV, when folded with an inverse beta decay cross section which increases with the square of the neutrino energy, yields an observed positron energy distribution which peaks in the vicinity of $20~$MeV.

High energy neutrino telescopes under construction deploy Optical Modules (OM's) in a clear medium which acts as the radiator of Čerenkov light from muon tracks or electromagnetic showers produced in neutrino interactions. They are primarily designed to exploit the long range of high energy muons and have typical nominal threshold energies in the GeV-range, ostensibly too high to observe supernova-neutrinos. However, the interactions of $\bar\nu_e$ with protons produces copious numbers of positrons with tens of MeVs of energy. These will yield signals in all OMs during the (typically 10 second) duration of the burst. Such a signal, even if statistically weak in a single OM, will become significant for a sufficient number of OMs. The number of OMs required for monitoring galactic supernovae has been shown to be of order a few hundred, a number typical for the first-generation detectors under construction. Apart from the excess signal and its time profile, additional information on the energy, direction or neutrino species is unlikely to be reconstructed in such telescopes, in contrast to dedicated supernova experiments.

The AMANDA-detector, operating in low background, sterile Antarctic ice, is at an advantage as far as detection of supernova bursts is concerned. The background counting rates in the OMs are dictated by the dark current of the photomultiplier only. They are reduced by over an order of magnitude compared to natural sea-water because of the absence of bioluminescence and radioactive decay of potassium. Recently, a second advantage of using polar ice has been revealed. With the first calibration measurements it has become evident that the absorption length of light for in-situ South Pole ice is larger than what had been anticipated. The absorption length of deep South Pole ice has the astonishingly large value of $\sim310$ m for the 350 to 400 nm light to which the PMTs are sensitive. A value of only 8 m had been anticipated from laboratory measurements. For supernovae neutrino signals the detector volume scales linearly with the absorption length. This can be demonstrated from a simple back-of-the-envelope estimate of the effective detection volume of an OM for MeV-positrons.

The track-length of a 20 MeV positron in ice is roughly 12 centimeters and therefore over 3000 Čerenkov photons are produced. This number combined with a typical quantum efficiency of $25~\%$ leaves $N_{\gamma} \simeq 800$ detectable photons in each event. At large distances the probability that the OM detects a photon falls as $R^{-2}$ $$P(R) \simeq (R_d / R)^2 \,\,\,\,\,\,\,\, {\rm for} \,\, R>>R_d$$ Here $R$ is the distance to the positron shower. $R_d$ is defined such that photon detection probability by the OM is essentially unity for distances smaller than $R_d$, and falls quadratically for larger distances. Its value can be estimated from the relation $$\pi R_d^2 \sim N_{\gamma} A,$$ where $A$ is the photocathode area of the OM. The relation states that the effective area for guaranteed detection is proportional to the brightness of the source and to the photocathode area. We next calculate the effective volume $V_{\rm eff}$ associated with each OM by integrating the probability function over volume: $$V_{\rm eff} \simeq \int_0^{R_{\rm abs}} P (R) R^2 dR \,.$$ The integral is, of course, cut off by the absorption length $R_{\rm abs}$ of the light. It is easy to see that its value is dominated by $R > R_d$ and negligible for $R < R_d$. Therefore $$V_{\rm eff}  \simeq \int_{R_d}^{R_{\rm abs}} P (R) R^2 dR
\sim {1 \over \pi} N_{\gamma} A R_{\rm abs}\,,$$ for $R_{\rm abs}\gg R_d$. This estimate can be made more quantitative, but is sufficient to demonstrate that the effective volume for supernova positrons, corresponding to each OM, is proportional to both the collection area of the OM and the attenuation length of the light. We will demonstrate that the discovery of very large values of $R_{\rm abs}$ for natural polar ice implies that the AMANDA detector can function as a galactic supernova watch.

At present, four AMANDA strings have been deployed in Antarctic ice at the South Pole. Each string holds 20 OMs separated by 10 meters at depths between 800 and 1000 meters. The calibration measurements have revealed absorption lengths of light for in-situ ice which are larger than what had been anticipated from laboratory experiments. The results have been obtained by studying the propagation of Čerenkov light from high energy muons, and by studying the propagation of laser light pulsed from small nylon spheres attached to the ends of an optical fiber below each OM. The results indicate that the propagation of photons in the ice is consistent with a diffusive process with very large absorption lengths. The diffusive character of the process apparently results from scattering of the light by residual bubbles trapped in the ice. The number density of such bubbles is observed to decrease with depth. Direct measurements on ice cores indicate that such bubbles totally disappear at 1–1.5 km depth, depending on location. Clearly these results have important implications for supernova neutrino detection because we expect a linear enhancement of the effective volume per OM with absorption length.

We have calculated the effective volume of the present AMANDA detector for supernova neutrinos, taking into account the increased attenuation length, and have also considered the effect of scattering of the light by residual air bubbles at 1 km depth. The above back-of-the-envelope estimate yields an effective volume $V_{\rm eff} \sim 2000$ m$^3$ for an AMANDA OM with collecting area $A=0.028$ m$^2$ and an absorption length $R_{\rm abs} \simeq 300$ m. The effective radius of each OM for detecting supernova neutrinos is about 7 meters. This is very encouraging and indicates that a single OM is similar in collection volume to the Kamiokande and IMB detectors.

We have confirmed and sharpened this result with a complete Monte Carlo simulation. Positron showers were generated using a 4 MeV Boltzman distribution multiplied by $E^2$ in order to represent the increased detection efficiency associated with the rise of the neutrino interaction cross section with energy. We used an electromagnetic shower Monte Carlo which was originally designed to calculate radio pulses in ice and correctly describes showers down to MeV energies. The output, in the form of Čerenkov photon distributions radiated by positron showers, was fed into a simulation of the AMANDA detector. This program treated in detail both the propagation of the photons, their absorption and scattering, and the efficiencies of the OMs. For the current AMANDA configuration, with 73 stable OMs, the simulations reveal an effective volume per OM $V_{\rm eff} \simeq 1.7
\pm 0.4~10^3~$m$^3$, compatible with our estimate and more than an order of magnitude above the results previously obtained assuming the properties of laboratory ice. The observable signal is totally dominated by events where a positron shower yields a single photoelectron hit in a single OM. The quoted error is statistical. Although the absorption length at the peak acceptance of the OMs is well measured, its dependence on color in the near UV region is poorly determined. More precise future measurements may increase or decrease our result but cannot modify it significantly.

We have verified by Monte Carlo that the effect of residual bubbles on photon propagation is within our statistical error. This is to be expected because our results imply that the photons are collected over a radius of 7.4 meters, a distance much shorter than the absorption length of the light. For a scattering length $R_s$ of order 0.25 meters photons within this radius will reach the PMT whether scattered or not. The scattering is irrelevant as long as the actual distance travelled by the scattered photons ($N R_s$, with N the number of times that the photon scatters), is shorter than $R_{\rm abs}$. For a random walk the number of scatters $N$ for a photon originating a distance d from the OM is given by $$N \simeq d^2 / R_s^2$$ The proportionality factor depends on the angular distribution of the scattering process; it is 4/3 in the present problem. The condition that the pathlength $N R_s < R_{\rm abs}$ becomes $$d < \sqrt{{4 \over 3} R_s R_{\rm abs}}$$ which is satisfied for all distances $d$ up to 7.4 meters for $R_s \simeq
0.25$ m and $R_{\rm abs} \simeq 310$ m.

These results can be used to rescale SN1987 observations to a supernova at a distance of $d_{\rm kpc}$. To rescale the data from the Kamiokande(IMB) experiment to the AMANDA detector we need the ratio of their effective volumes corrected for the different thresholds of the instruments. This introduces a correction factor $f$ which is evaluated from the Fermi-Dirac energy distribution and the $E^2$-dependence of the cross section. Using a supernova temperature for SN1987A of $4.0$ MeV $f$ is about 0.8 (0.2) for Kamiokande (IMB) because of the reduced threshold of AMANDA operating as a pure counting detector. From the 11 events observed in the 2.14 kton Kamiokande detector we predict:

$$N_{\rm Events} \sim 11~N_M~\left[{\rho~V_{\rm eff} \over f~2.14~{\rm kton}}
\right]
\left[ {52~{\rm kpc} \over d_{\rm kpc}} \right]^2$$ For the present AMANDA configuration with $N_M=73$, $\rho=0.924$ g cm$^{-3}$, the density of ice, and $V_{\rm eff}=1700$ m$^3$, we obtain 270 events per OM for a total of nearly 20,000 events.

We must now require a meaningful detection of this signal in the presence of the continuous background counting rate of all phototubes. Over the 10 seconds duration of the neutrino burst, the RMS-fluctuations of the combined noise from all the OMs is: $$\sigma_{1p.e.} = {\sqrt {(10\rm~sec) \nu_{1p.e.}~N_M }}.$$ Here $\nu_{1p.e.}$ represents the average background counting rate in a single module at the 1 photoelectron level, and equals1.86 kHz for the AMANDA OMs. The probability that this noise fakes a supernova signal can be, at least theoretically, estimated from Poisson statistics. The expected rate of supernova explosions in our galaxy is about $2\times10^{-2}$ y$^{-1}$. If the detector is to perform as a supernova watch we must require that the frequency of fake signals is well below this rate. The signal should therefore exceed $n_{\sigma} \ge 6$ which corresponds to a probability of $9.9\times10^{-10}$. The corresponding number of 10 second intervals does indeed exceed a century. This requirement can be relaxed if we just demand that the detector can make a measurement in the presence of independent confirmation. This may be achieved by operating parts of a single instrument as independent detectors.

For an average noise rate of 1.86 kHz the RMS fluctuation of the 1.36 million hits expected in an interval of ten seconds is 1165. This yields a detected excess of 17 $\sigma$ for a supernova at the center of the galaxy. Demanding a 6 $\sigma$ signal for a supernova watch, the present AMANDA configuration with appropriate triggering, would view a radius of 17 kpc, thus covering our galaxy. Clearly signal-to-noise is proportional to $\sqrt{N_M}$ and therefore the number of OMs needed for fixed signal-to-noise ratio scales with the distance to the fourth power. A next-generation detector with 6000 OM's would provide a sharp signal for a supernova within a sphere of radius 52 kpc.

The results are extremely encouraging and an effort should be made to find the optimal trigger. In a realistic detector the background counting rate does not follow a Poisson distribution with a width given by simple statistics. The actual performance of the present AMANDA data acquisition relevant to the detection of supernovae is described elsewhere. The AMANDA detector, frozen in then Antarctic ice, can act for decades as a supernova watch with minimal maintenance or investment in manpower and operation of the instrument.

This work was supported in part by the University of Wisconsin Research Committee with funds granted by the Wisconsin Alumni Research Foundation, in part by the U.S. Department of Energy under Grant No. DE-FG02-95ER40896, in part by the Xunta de Galicia under contract XUGA-20604A93 and in part by and the CICYT under contract AEN93-0729.