Learning Policies through Quantile Regression
Oliver Richter Roger Wattenhofer

 

Factorization problems in complex reflection groups
    Joel Brewster Lewis and Alejandro H. Morales
    
===================================================




Policy gradient based reinforcement learning algorithms coupled with neural networks have shown success in learning complex policies in the model free continuous action space control setting. However, explicitly parameterized policies are limited by the scope of the chosen parametric probability distribution. We show that alternatively to the likelihood based policy gradient, a related objective can be optimized through advantage weighted quantile regression. Our approach models the policy implicitly in the network, which gives the agent the freedom to approximate any distribution in each action dimension, not limiting its capabilities to the commonly used unimodal Gaussian parameterization. This broader spectrum of policies makes our algorithm suitable for problems where Gaussian policies cannot fit the optimal policy. Moreover, our results on the MuJoCo physics simulator benchmarks are comparable or superior to state-of-the-art on-policy methods.




§ INTRODUCTION




Over the years, algorithms in artificial intelligence have become less restrictive in their assumptions, going from expert systems to kernel engineering to deep learning approaches, making the same algorithms applicable to a wider range of problems. In this mindset, we start from one of the least restrictive learning settings - continuous action space deep reinforcement learning - and show that removing the restriction of an explicitly parameterized policy can in fact improve performance. 
To achieve this, we cut loose of policy gradient methods and show that policies can be improved by moving probability mass through quantile regression towards adequate actions. 

The core idea of our algorithm is to use quantile regression <cit.> for an implicit approximation of the optimal policy in the network parameters. Modeling the policy implicitly in the network allows us to approximate any distribution, i.e., we are not limited to an a-priori defined parameterized distribution (like multivariate Gaussian). This allows us to learn complex, multi-modal, non-Gaussian policies that are automatically inferred from interaction with the environment. While under full observability there exists an optimal deterministic policy <cit.> this is not the case in multi-agent and limited history setups. Here, stochastic, multi-modal policies are required to reach optimality; a complication that is often overlooked when applying an out-of-the-box Gaussian policy. On top of the stochastic flexibility we show that our algorithm has a trust region <cit.> interpretation and also achieves good performance on a large set of challenging control tasks from the OpenAI gym benchmark suite <cit.>.

To summarize, our contributions are:

    
  * A derivation of a new reinforcement learning algorithm for continuous action space policies
    
  * A discussion showing formal similarity between trust region policy gradient methods and quantile regression deep reinforcement policy learning
    
  * An empirical evaluation showing the flexibility and competitiveness of the proposed algorithm
    
  * And as a byproduct: A novel neural architecture for monotonic function approximation




§ BACKGROUND AND RELATED WORK


In reinforcement learning <cit.> an agent tries to learn a task through interaction with the environment in which the task is defined by means of a reward signal. More formally, in each time step t the agent chooses an action a_t based on the current state _t∈ and gets as feedback the reward r_t and next state _t+1 from the environment.  denotes the set of all possible states. The goal of the agent is to maximize the discounted cumulative reward R=∑_t γ^t r_t, with discount factor γ∈ [0,1], by adapting its policy to more rewarding trajectories. For our derivation, we will use the standard definitions <cit.> of value function V_π(_t) and action-value function Q_π(a_t,_t)

    V_π(_t) = _π|_t[∑_t'=t^∞γ^t'-t r_t']


    Q_π(a_t, _t) = __t+1|a_t,_t[r_t + γ V_π(_t+1)]

where the expectation in V_π is taken over trajectories collected when acting according to policy π starting from state _t. We 

use 

the star symbol (as in π^*) to denote optimality.

There are two main approaches in the literature to achieve reinforcement learning: Value based methods, foremost Q-learning <cit.>, and policy gradient based methods <cit.>.
Both methods have been adapted to use deep neural networks as function approximators <cit.> where policy gradient based methods are often combined with a critic into actor-critic methods to reduce the variance in the gradients. Deep reinforcement learning, the combination of deep learning and reinforcement learning, has also found its way into the continuous action space setting <cit.>. Especially actor-critic methods with proximal second order updates <cit.> - which ensure that the policy stays within a trust region - have shown success in learning complex policies in the continuous action space setting. A now popular algorithm, called Proximal Policy Optimization (PPO), was introduced by  PPO, who showed that the trust region performance boost can be achieved by optimizing a clipped first order objective.

Recently, several papers <cit.> showed that soft Q-learning is in fact equivalent to maximum entropy actor-critic methods; a setup in which the agent is encouraged to maximize the entropy of its policy besides maximizing the reward. Combining a stochastic actor with the sample efficiency of Q-learning <cit.> has led to impressive results.  latent also extended this setup to a hierarchy of latent policies, where higher level policies control lower level actions through an invertible, normalizing flows network <cit.>.  boostnormflow showed that a normalizing flows network can boost the performance of on-policy continuous action space algorithms as well. The drawback of normalizing flow networks is, however, that the network needs to be invertible and thereby requires the computation of the determinant of the Jacobian in each layer for the probability mass propagation. This limits normalizing flows in practice to narrow networks, a limitation not faced by quantile networks as in our setup.

One of the early works on maximum entropy deep reinforcement learning <cit.> however already showed that one can train a particle based actor to approximate the (possibly multimodal) maximum-entropy policy defined by the soft Q-function with Stein-variational gradient descent <cit.>. This work was extended by  WGF which showed that policy optimization can be seen as Wasserstein Gradient Flow in the policy probability space with the 2-Wasserstein distance as metric. This led them to propose an additional loss on the action particles which improved performance. The relationship between policy optimization and Wasserstein Gradient Flows was simultaneously shown by  WGF_. The problem with particle based methods is, however, that for a general sampling network the likelihood of a given sample cannot be directly recovered. Therefore, off-policy corrections for multi-step learning as proposed in <cit.> cannot be applied. As we discuss 

in Appendix <ref>, the sample likelihood can be retrieved from the quantile function. Our method could therefore in principle be extended with these off-policy corrections in future work.

 VIREL recently introduced a variational inference framework for deep reinforcement learning. They investigated the probabilistic nature of policies learned by deep reinforcement learning algorithms from a variational inference perspective, but revert to Gaussian policies in their experiments for implementation convenience.

 QR-DQN,IQN were the first to use quantile regression in connection with deep reinforcement learning. In their work,

they focused on
approximating the full probability distribution of the value function. In contrast, we
explore possibilities of using quantile regression to approximate richer policies by not constraining the action distribution to an explicitly parameterized distribution.  QN_generative_modeling showed that quantile networks can also be used for generative modeling while recently  quantile_forecasting used quantile regression in combination with a recurrent neural network for probabilistic forecasting. In general, we see quantile regression in combination with deep learning to have a lot of potential for future work as it provides a flexible way to represent an arbitrary probability distribution within the network.



§ QUANTILE REGRESSION AND QUANTILE NETWORKS



Quantile regression <cit.> discusses approximation techniques for the inverse cumulative distribution function F_Y^-1, i.e., the quantile function, of some probability distribution Y. Recent work <cit.> showed that a neural network can learn to approximate the quantile function by mapping a uniformly sampled quantile target τ∼𝒰([0,1]) to its corresponding quantile function value F_Y^-1(τ)∈. Thereby, the trained neural network implicitly models the full probability distribution Y. 

More formally, let W_p(U,Y) be the p-Wasserstein metric

    W_p(U,Y) = (∫_0^1 |F_Y^-1(ω) - F_U^-1(ω)|^p dω)^1/p

of distributions U and Y, also characterized as the L^p metric of quantile functions <cit.>.  QR-DQN show that minimizing the quantile regression loss <cit.>

    ρ_τ(δ) = (τ - _δ < 0)·δ


reduces the 1-Wasserstein distance between a scalar target probability distribution Y and a parameterized mixture of Diracs U_θ, which correspond to fixed quantiles τ. Here, δ = y - u with y∼ Y and u = F_U^-1(τ) is the quantile sample error.  QN_generative_modeling generalized this result by showing that the expected quantile loss

    ℒ = _τ∼𝒰([0,1])[_z∼ Z[ρ_τ(z - Ĝ_θ(τ))]]

of a parameterized quantile function Ĝ_θ aproximating the quantile function F_Z^-1 of some distribution Z is equal to the quantile divergence


    q(Z,π_θ) := ∫_0^1[∫_F_Z^-1(τ)^F_π_θ^-1(τ)(F_Z(x)-τ)dx]dτ


plus some constant not depending on the parameters θ. Here, π_θ is the distribution implicitly defined by Ĝ_θ. Therefore, training a neural network Ĝ_θ(τ) to minimize ρ_τ(z-Ĝ_θ(τ)) with z sampled from the target probability distribution Z effectively minimizes the quantile divergence q(Z,π_θ) and thereby models an approximate distribution π_θ of Z implicitly in the network parameters θ of the neural network Ĝ_θ(τ). 

Another way to state this result is by noting that the quantile regression loss ρ_τ appears in the continuous ranked probability score (CRPS) <cit.>

    CRPS(F^-1, z) := ∫_0^1 2 ρ_τ(z - F^-1(τ)) dτ

which is a proper scoring rule <cit.>, i.e., _z∼ Z[CRPS(F_Z^-1, z)] ≤_z∼ Z[CRPS(F_Y^-1, z)] for any distributions Z and Y. Minimizing the expected quantile loss (<ref>) is equivalent to minimizing the expected score _z∼ Z[CRPS(Ĝ_θ, z)] which leads to Ĝ_θ approximating F_Z^-1 (derivation adapted from  quantile_forecasting). Note that this derivation requires Ĝ_θ to always define a proper quantile function, which we ensure in this work by modeling Ĝ_θ through a monotonically non-decreasing neural network.

By approximating the quantile function instead of a parameterized probability distribution, as it is common in many deep learning models <cit.>, we do not enforce any constraint on the probability distribution Z. E.g., Z can be multi-modal, i.e., non-Gaussian, as a strong function approximator, like a neural network, can approximate the corresponding quantile function.



§ QUANTILE REGRESSION REINFORCEMENT LEARNING
 

Given that a quantile network can approximate any probability distribution, we aim at approximating the optimal policy in a reinforcement learning setup. For this, we model for each action dimension the quantile function Ĝ_θ(τ, ) = F_π_θ^-1(τ, ) of the implicitly defined action distribution π_θ by a neural network with a state  and a target quantile τ∈ [0,1] as input. In practice, we share the first few layers extracting features from the state into an embedding ϕ(), before we pass the embedding to the individual quantile networks Ĝ_θ(τ, ϕ()). However, for ease of notation we omit this shared network ϕ(·) in the following derivations.[See Appendix <ref> for more details. Our exact implementation can be found in the supplementary material.] From the full network Ĝ_θ: [0,1]^d×→^d, with d being the number of action dimensions, an action ∈≡^d for a given state  can be obtained by sampling τ∼𝒰([0,1]^d) and taking the network output as action. Since the network approximates quantile functions, the network output of a uniformly at random sampled quantile target is a sample from the implicitly defined action distribution. The question left to address is how to train the network, such that it (a) represents the quantile functions of the action dimensions and (b) the implicitly defined policy maximizes the expected (discounted) reward R.

Objective (a) can be achieved by limiting the quantile network to a monotonic function with respect to the quantile input τ. This ensures that the network represents a valid quantile function already at the start of training. Note that such an architectural prior, although sensible, was not applied in related work <cit.>. Here however, it is vital, since it allows us to perform gradient ascent on the quantile loss

(detailed below), as the network remains a valid representation of a quantile function. A monotonically non-decreasing neural network can be achieved by restricting the neuron connecting weights to be positive and applying a non-linearity that allows for concavity as well as convexity. For weight positivity we simply set the network weights θ^W_net of the quantile net to the element-wise exponent of unconstrained weights θ^W, i.e., θ^W_net = expθ^W. As non-linearity, we choose a combination of ReLUs, with neuron output y=max(0,x) given neuron input x, and inverse ReLUs, where y=min(0,x).  We detail and contrast our architecture design against other designs in
Appendix <ref>, extending the current literature on monotonic networks.


The more difficult objective is (b); achieving a policy improvement over time. Here we address this with advantage weighted quantile regression. Informally put, quantile regression is linked to the Wasserstein metric which is also sometimes referred to as earth mover's distance. Imagine a pile of earth representing probability mass. In reinforcement learning we essentially want to move probability mass towards actions that were good and away from actions that were bad, where good and bad are measured by discounted cumulative (bootstrapped) reward received. Quantile regression can achieve this

by shaping the pile of earth according to an advantage estimation and the constraint of monotonicity

(a core property of quantile functions).

More formally, we are interested in approximating the optimal policy π^*(a), where we omit the implicit dependence on the state  in the following for ease of readability.[We focus our analysis on the simple case of a single action dimension with scalar input and scalar output. The generalization to multidimensional independent action-/quantile-functions follows trivially.]

If we were given the optimal policy, this could be achieved by training on the quantile regression objective

    θ^* = _θ_a∼π^*[_τ∼𝒰([0,1])[ρ_τ(a - Ĝ_θ(τ))]]

as this would minimize the quantile divergence between π^* and π_θ as derived in the previous section. However, sampling from the optimal policy π^* is infeasible, since we do not know it a-priori.
Therefore we rewrite our objective as

    θ^* = _θ_a∼μ[_τ∼𝒰([0,1])[π^*(a)/μ(a)ρ_τ(a - Ĝ_θ(τ))]]

where μ is a policy with support greater or equal to the support of π^*. The importance ratio π^*(a)/μ(a) gives a measure of how much more/less likely a given action a would be under the optimal policy π^* compared to the policy μ that collected the experience. 
To ease our calculations, we assume the behaviour policy μ to be uniform, i.e., to have constant likelihood μ(a) for all actions a within the support of μ. This lets us absorb the experience likelihood μ(a) into the optimization.



Appendix <ref> discusses this approximation further.


Following <cit.> we define the optimal soft policy at temperature β as π_β^*(a)∝exp(Q^*(a) - V^*/β), where Q^* and V^* are the soft action-value and value function of the optimal policy. We choose the soft formulation to retain a stochastic policy, as we are interested in setups, where the optimal policy is not deterministic.

We can then write our objective as

    θ^* = _θ_a∼μ[_τ∼𝒰([0,1])[ℒ_θ(a,τ)]]


    with  ℒ_θ(a,τ) = exp(
        Q^*(a) - V^*/β)ρ_τ(a - Ĝ_θ(τ))


This shifts the perspective from estimating the likelihood under the optimal policy to estimating the advantage A^*(a) = Q^*(a) - V^* of the optimal policy over the (sub-optimally) chosen action a∼μ. While this quantity is still unknown a-priori, we can approximate the advantage of our current policy π_θ. If we therefore approximate the optimal soft policy by the exponential of A_π_θ, the advantage of our current best policy, we arrive at the iterative optimization procedure


    θ_k+1 = _θ_a∼μ[_τ∼𝒰([0,1])[ℒ_θ^k(a,τ)]]


    with  ℒ_θ^k(a,τ) = exp(
        A_θ_k(a)/β)ρ_τ(a - Ĝ_θ(τ))

where we used the short notation A_θ_k to denote the advantage of policy π_θ_k.
Note that this objective has a (stochastic) convergence fixpoint with π_θ≡π_β^*.
In practice however, it is cumbersome to define a behaviour policy μ for experience collection a-priori, because an a-priori defined μ might not cover the support of the optimal policy π_β^*, as required by the derivation. On the other hand, an a-priori defined μ that does cover the support of π_β^* might be too explorative, rendering only a few informative action samples, which would result in a low sample efficiency of the algorithm. What we would therefore like to do is to use our current best estimate of the optimal policy to gather experience. Note, however, that the optimization (<ref>) contracts π_θ onto the support of μ, since the objective regresses π_θ towards samples of μ. If we set μ≡π_θ and use an approximate iterative procedure like stochastic gradient descent, at some point the support of π_θ would not cover the support of π^* anymore and the policy would degenerate in the limit towards a sub-optimal deterministic action in each state. Heuristics, such as entropy regularization <cit.>, have been shown to circumvent similar problems in the discrete action space setup. Note however, that entropy regularization would be non-trivial in our quantile function continuous action space setup. Instead, we propose the following: we take a linear approximation to the exponential function exp(x)≈ x + 1 around x≈0 and multiply by β to get the regularized iterative objective function

    ℒ_θ^k(a,τ) ≈ A_θ_k(a)ρ_τ(a - Ĝ_θ(τ)) + βρ_τ(a - Ĝ_θ(τ))

Note that this approximation is reasonable for A_θ_k(a)≈ 0, which in turn is reasonable for a∼π_θ_k given a decent approximation of the advantage. This linearization however brings an interesting property: if an action taken by the behaviour policy results in an outcome that is worse than the outcome predicted for π_θ_k, i.e., the advantage A_θ_k(a) is negative, the objective regularizes in that we maximize the corresponding quantile loss ρ_τ. This essentially pushes action samples a'∼π_θ away from the bad action a, thereby expanding the support of π_θ. Note that ascending on the quantile loss does not lead to divergence since we restrict our quantile network to be monotonically non-decreasing
and therefore any parametarization of the network results in a valid quantile function. 
We can therefore now safely replace the behaviour policy μ to generate experience with our current best estimate π_θ_k. This gives us an effective, iterative on-policy algorithm, that regresses towards the optimal soft policy based on experience, while regularizing against a premature collapse to a local optimum.




















































Another motivation for objective (<ref>) is its similarity to trust region methods <cit.>: while the first part in objective (<ref>), A_θ_k(a)ρ_τ(a - Ĝ_θ(τ)), gives a measure for how much we will update our policy based on the current rollout, the second part, βρ_τ(a - Ĝ_θ(τ)), constrains the updated policy to not deviate too far from the behaviour policy which gathered the experience. I.e., the second term defines a trust region objective, which is controlled by β. A high value for β keeps the policy close to the policy that collected the current batch of experience, yielding update stability, while a low value for β allows the policy to adjust more rapidly to new experiences. We use this similarity of our algorithm to trust region methods to take advantage of algorithmic improvements employed by  PPO and adapt them without adjustment in our algorithm. Namely, in our reinforcement learning experiments we use generalized advantage estimation <cit.> to approximate A_θ_k(a), normalize the advantages 

and train a small number of epochs with mini-batches on each collected rollout. Pseudo code is available in Appendix <ref>.

Further, note that the number of samples taken from the inner expectation in objective (<ref>) can be 

chosen to trade-off required computation against gradient variance. In some initial experiments on the MuJoCo Swimmer and Ant task we varied the number of samples K∈{1,32,128,256} and chose K=128 as we found this to be a good trade-off. Similarly we chose β=2 from {0,1,2,4} based on the same tasks. The remaining hyper-parameters were copied without adjustment from  PPO and are reported in Appendix <ref>. The fact that our algorithm works without any adjustment of these hyper-parameters hints at its robustness and ease of applicability. We leave it to future work to find a better 

hyper-parameter set.

















§ EXPERIMENTS AND RESULTS



As recent papers have raised valid concerns about the reproducibility of deep reinforcement learning results in the continuous action domain <cit.>, we ran all our experiments for 20 fixed random seeds (0-19). Implementation details and hyper-parameter choices to reproduce the results can be found in the supplementary material. With our experiments we aim to answer the following questions:


    
  * Is the ability to learn more diverse probability distributions worthwhile to get an advantage in simple two player games? Do multimodal policies emerge from training on these games?
    
  * How does quantile regression deep reinforcement learning (or QRDRL in short) compare to other online algorithms on well studied reinforcement learning benchmarks?


As a first experiment, we implemented a continuous action version of Rock-Paper-Scissors. Rock-Paper-Scissors can be seen as a two player multi-arm bandit problem, each player chooses one of the actions Rock, Paper or Scissors and the winner is determined based on the choices: Rock beats Scissors, Scissors beats Paper and Paper beats Rock. We modeled the players through algorithms choosing their action in a 1-dimensional continuous action space, where we defined intervals for the corresponding discrete actions as shown in Figure <ref>. We aim to learn a non-exploitable policy in this setup, that is, we train a policy such that a countering policy, trained to exploit the former policy, achieves the minimum possible wins. Specifically, in each training iteration we train a countering Gaussian policy from scratch on 10,000 games against the current policy and then use this countering policy as opponent in 100 games based on which the current policy is updated.



We trained two policy networks in this setup: (i) a quantile network trained on the weighted quantile loss _a∼π_θ[_τ∼𝒰([0,1])[r·ρ_τ(a - Ĝ_θ(τ))]], where the weight r=1 for games that were won and r=-1 for games that were lost,[This is the multi-arm bandit analogy to (<ref>) with β=0.] and (ii) a Gaussian policy trained to maximize the log-likelihood of winning, i.e., maximize r, via policy gradient. The exact experiment setup is described in Appendix <ref>.

Figure <ref> shows histograms of the action distributions learned by the two approaches while Figure <ref> shows the average return r throughout the training. The results show that the uni-modal nature of the Gaussian network can always be exploited by the countering policy, hindering any learning progress. On the other hand, the Quantile network learns to choose close to uniform at random, making the policy impossible to exploit. Moreover, it learned that the countering Gaussian policy is initialized with the mean close to 0 - Paper - which explains the slight tilt of the action distribution towards the right - Scissors - and the slightly above zero return at the end of the training. I.e., it has learned to exploit the initialization and inability to counter of the countering policy.

However, we did not find the multi-modal nature in the learned action distribution that we were hoping for. To verify that our approach can indeed learn a multi-modal policy we implemented another toy game. In this game, which we call Choice, the agent also acts in a single continuous action dimension, where an action between -0.6 and -0.4 corresponds to a button A pressed while an action between 0.4 and 0.6 corresponds to a button B pressed. The agent is rewarded if it presses the button that was pressed less often so far within the episode. The problem is complicated in that we only model the agent as a feed-forward network, giving it no ability to remember the actions it took so far. Note that in this setup the best policy is to choose one of the two buttons at random. Implementation details can be found in Appendix <ref>.










As can be seen in Figure <ref>, QRDRL is capable of recovering the two modes needed to solve the task while proximal policy optimization (PPO,  PPO), a commonly used Gaussian policy gradient method, learns a suboptimal compromise between the two buttons. This is especially apparent when we look at the corresponding return throughout the training depicted in Figure <ref>: the return of PPO stagnates around 1.5 while QRDRL continues to improve throughout the training. Nevertheless, QRDRL was unable to put 0 probability on the invalid actions between the modes within the given training time. We believe this stems mainly from our architecture choice and the artificial discontinuous distribution setup that is difficult to approximate (see Appendix <ref>). 


Given that the ability to express more complex stochastic policies is indeed vital to perform well in the toy games presented, we are left with our second research question, whether QRDRL also performs well on commonly used reinforcement learning benchmarks. To this end, we run our algorithm on a diverse set of robotic tasks defined in OpenAI gym <cit.> based on the MuJoCo <cit.> physics simulator. These include diverse robotic figures which should learn to walk as well as robotic arms and pendulums which need to reach a certain point or balance themselves. We compare our approach against PPO <cit.> as well as normalizing flows TRPO <cit.>, a recently proposed on-policy algorithm using normalizing flows.[We compare to their TRPO version since their normalizing flow ACKTR implementation yielded worse/inconsistent results.] Implementation details are provided in Appendix <ref>





The results reported in Figure <ref> show that QRDRL is indeed capable of performing as well as, if not superior to current state-of-the-art on-policy methods. Specifically, we clearly outperform these methods on the pendulum tasks as well as the challenging Ant task. We see a similar trend in the normalizing flows TRPO (N-TRPO) results, suggesting that a more flexible policy is especially helpful in these tasks. However, N-TRPO fails to match our results in many cases, which we believe might be due to the limitation of a narrow flow network. Note also that we achieved our results without extensive hyper-parameter tuning, and our results might even improve for better suited hyper-parameters. 





§ CONCLUSION AND OUTLOOK


In this work, we introduce quantile regression deep reinforcement learning (QRDRL), a likelihood free reinforcement learning approach to overcome the limitation of explicitly parameterized policies. We show that the ability to learn more diverse stochastic continuous action space policies is essential to perform well in different situations, be it to defer the exploitability of a policy or to cope with memory limitations. We further show that our algorithm has performance comparable to state-of-the-art on-policy methods on the well studied MuJoCo benchmarks.

Given that this work introduces a new idea to approach reinforcement learning with quantile regression, it offers many directions for future work:

Off-policy learning: We showed in this work how quantile regression can be used within an on-policy algorithm. Future work could extend our approach to the off-policy setting, leveraging the advantage of sample reuse. Note that while our training objective is likelihood free, we can still recover the action likelihood through a simple gradient back propagation (see Appendix <ref>). This is a distinct advantage of our approach over other particle based algorithms, as it enables the potential use of multi-step off policy corrections as described by  Retrace or  impala.

Multivariate quantile regression: We took a conservative approach in this work by modeling each action dimension as an independent quantile function. This is along the line of Gaussian policy implementations, where off-diagonal elements of the covariance matrix are omitted as well. However, insights into multivariate quantile regression <cit.> could be adapted to give our algorithm the capability to infer the stochastic relations between action dimensions.







aaai.bst






 


§ LIKELIHOOD IN QUANTILE FUNCTIONS


While there are different ways to approximate an arbitrary probability distribution Y, we want to highlight a useful trait of approximating the quantile function.
Specifically, we note that the partial derivative of the quantile function F^-1_Y(τ) with respect to the quantile τ - also known as the sparsity function <cit.> or quantile-density function <cit.> - has the interesting property <cit.>:

    δ/δτF^-1_Y(τ) = 1/p_Y(F^-1_Y(τ))

where p_Y(·) is the probability density function of the distribution Y. In our setup, this allows us to retrieve the likelihood of an action a_τ = Ĝ_θ(τ) as

    π_θ(a_τ) = 1/δ/δτĜ_θ(τ)

where π_θ is the probability density function of the policy distribution implicitly defined in our quantile function approximation Ĝ_θ.



§ MONOTONIC NEURAL NETWORKS


While there are several methods for monotonic function approximation in the literature <cit.>, (partially) monotonic neural networks <cit.> are mainly constructed using either the max-pool, min-pool architecture introduced by  monotonic_net or a positive weight network with tanh-activations as introduced by  tanh_mono_net. Other approaches include training a monotonic network through genetic algorithms <cit.> or using a mixture of monotonic networks for prediction <cit.>. More recently,  lattice_mono_net introduced a monotonic neural network with multidimensional lattices as non-linearities. We believe that this approach over-complicates the problem of monotonic function approximation with a neural network and restrain to a novel, simple yet effective way.

First of all, we note that a neural network is a combination of functions, most commonly a combination of linear embeddings (i.e., matrix multiplication) and element-wise non-linearities (e.g., tanh or ReLU). The easiest way to restrict a neural network to approximate only monotonic functions is therefore to restrict all constituent functions to be monotonic with respect to their inputs. This can be done by restricting the matrix multiplications to have only positive weights and use monotonic element-wise non-linearities, as was done with tanh-activation by  tanh_mono_net. 
Weight positivity is easily achieved by setting all parameters of the network weight matrices θ^W_net to the exponent of unconstrained weights θ^W, i.e., θ^W_net = exp(θ^W) with exp(·) applied element-wise, as was done by  monotonic_net. We can then optimize θ^W using standard stochastic gradient decent methods. In our experiments we initialize θ^W to
log(𝒰([0,√(σ/F_in)]))
where F_in is the number of neurons in the previous layer and σ is a hyperparameter we set to 3. Note that this positivity constraint through exponentiation is only done for the weights and not for the biases, since the biases do not influence the monotinicity.

The more interesting architecture choice is the choice of non-linearity. In our work, we take advantage of the good performance reported for ReLU non-linearities <cit.>. Note that ReLU activation, y = max(0, x), is monotone with respect to its input. More so, it is convex, which would lead to a network only capable of approximating convex functions, if we restrict the network weights to be positive. To overcome this limitation, we use an adjusted ReLU activation on the hidden layers of our monotonic neural network. That is, we take half of the embedding dimensions and apply a ReLU activation, while we apply an inverse ReLU activation, y = min(0, x), to the remaining embedding dimensions. Note that the inverse ReLU activation is concave and a following linear layer with positive weight matrix can therefore make any combination of concave and convex functions, giving it the capability to approximate any monotonic function for a large enough hidden embedding.



To compare our architecture against the max-pool/min-pool architecture of  monotonic_net and an architecture with tanh-non-linearity as proposed by  tanh_mono_net, we train simple one hidden layer networks of each architecture with matched number of parameters (64 hidden units for the ReLU and tanh architecture, 96 hidden units for the Max-Min architecture) to approximate the quantile function of some toy distributions based on a scalar quantile input τ. τ is fed to the network as one dimensional input, sampled from a (scaled and shifted) uniform distribution over [-1, 1]. As 1-dimensional target distributions we choose a normal Gaussian (mean 0 and standard deviation 1), a bi-modal gaussian distribution (two Gaussians with mean -1 and 1 and both with standard deviation 0.5; a sample is drawn with equal probability from either of the two) as well as a discontinuous uniform distribution (samples are drawn with equal probability either uniformly from [-1,-0.5] or uniformly from [0.5, 1]). We train each network on 10,000 mini-batches of 128 samples each on the quantile regression loss (<ref>). The networks are trained on the learning rates {0.00001, 0.0001, 0.001, 0.01, 0.1} and the best results achieved (averaged over 5 random seeds) are compared in Table <ref>.[Code to re-run this experiment is available in the supplementary material.]



We note that while no architecture is significantly better than the others, we see a tendency of our ReLU architecture to improve in normal distribution approximation. 
Further, note that the Max-Min architecture requires comparatively large learning rates. We believe this is due to the fact that only a few parameters are updated with every sample (due to the pooling), therefore requiring a larger learning rate to be competitive with the other architectures. Lastly, we further investigate the bad performance of our ReLU architecture on the discontinuous uniform distribution task. In Figure <ref> we see that the ReLU network has difficulties to approximate the discontinuity. We believe that the stochastic nature of the task prohibits any weight in the network to grow large enough to approximate the discontinuity with a steep slope within the limited training time. Note that the other architectures provide inductive biases that help to deal with the discontinuity.

Nevertheless, we chose our ReLU architecture for our reinforcement learning agent, since we believe that the ability to approximate (multi-modal) normal distributions well out-weights the ability to approximate discontinuities well. Note that we might get even better results in our toy experiments if we had chosen otherwise.





§ CONSTANT ACTION LIKELIHOOD Μ(A)


In our derivation in Section <ref> we initially assume the behaviour policy μ to be uniform, such that the action likelihood μ(a) gets absorbed as constant in the optimization. Later we then set μ≡π_θ_k, which is clearly a stretch on the assumption of uniformity, especially if we try to approximate multi-modal distributions with π_θ_k. However, in early experiments we found that π_θ_k stays often close to a uniform distribution and mainly adjusts the support. Assuming the distribution goes from a brought, stochastic to a less stochastic distribution with smaller support, then the corresponding (constant) likelihood increases accordingly. This translates in a decreased learning rate over time (as the objective is multiplied by an ever smaller constant), which might also help the algorithm converge. We therefore retained the constant action likelihood approximation μ(a)≈const. . Note however, that theoretically one can calculate the action likelihood based on the gradient of the action with respect to the quantile (see Appendix <ref>), store it alongside the action in the rollout data and correct the optimization objective accordingly. We leave the investigation of such a correction to future work, as action likelihoods in continuous action spaces can be non-trivial to handle.



§ PSEUDO CODE


A pseudo code version of our proposed algorithm can be seen in Algorithm <ref>.






§ ROCK-PAPER-SCISSORS


In our Rock-Paper-Scissors experiment we compare a Gaussian policy network against a quantile policy network. Both policies are trained as described in the main text against a Gaussian countering policy with the same specifications as the other Gaussian policy network detailed hereafter. The policies opposing each other in a game choose simultaneously an action and the winner is determined as follows:

    
  * If both policies chose a valid action, i.e., an action within one of the three action ranges depicted in Figure <ref>, the winner is determined based on Rock-Paper-Scissors rules. That is, Rock wins over Scissors, Paper over Rock and Scissors over paper. If both policies chose the same action, the game results in a draw (r=0)
    
  * If one of the policies chose an invalid action, i.e., an action outside the three action ranges, the other policy wins with any valid action.
    
  * If both policies chose an invalid action, the game results in a draw.


As Gaussian policy network, we implemented a simple 2 layer fully connected neural network, which takes the two actions played in the last game as input, projects them to a hidden layer with 64 neurons and ReLU activation, and then back to a 2-dimensional embedding, from which the first dimension is taken as mean and the second dimension as log-standard deviation of the Gaussian defining the action distribution. 

Our quantile policy network also consists of 2 fully connected layers, where the first takes as input the one dimensional (scaled and shifted) quantile, sampled from a uniform distribution over [-1, 1], and projects it through a positive weight matrix to the hidden representation with 64 neurons. Half of the neurons in the hidden representation have ReLU activation, while the other half goes through an inverse ReLU non-linearity, as described in Appendix <ref>. The hidden representation is then projected with a positive weight matrix to a single output dimension, directly representing the action.
Both, Gaussian and quantile network, have trainable, unconstrained bias terms added in each layer. Also, we did not use multiple samples for the quantile loss in this experiment, i.e., we set K=1.

Note that the Gaussian network has (i) more parameters than the quantile network and (ii) more information given to it through the input: The Gaussian network can base its action on the last action played, giving it the ability to exploit any deterministic policy. Nevertheless, the quantile network can better fit the action space, leading to better results (see main text).



§ REINFORCEMENT LEARNING EXPERIMENTS



For our reinforcement learning experiments, the toy Choice game as well as the MuJoCo experiments, we adjusted the PPO implementation published in the OpenAI baselines github repository: https://github.com/openai/baselines

That is, our policy network consists of a state feature extraction part equivalent to the PPO network architecture before the final projection to the Gaussian parameters. We replace this final projection through a quantile net per action dimension similar to the one described in Appendix <ref> with the only difference that we add the extracted features to the hidden representations of the quantile networks. We summarize the hyperparameters used in Table <ref>. For our baselines, we used the implementations provided by the authors.






