DiGrad: Multi-Task Reinforcement Learning with Shared Actions
Parijat Dewangan S Phaniteja K Madhava Krishna Abhishek Sarkar Balaraman Ravindran


Computational study on microstructure evolution and magnetic property of laser additively manufactured magnetic materials
    Min Yi Bai-Xiang Xu Oliver Gutfleisch
    Received: July 10, 2018 / Accepted: 
=========================================================================================================================




Most reinforcement learning algorithms are inefficient for learning multiple tasks in complex robotic systems, where different tasks share a set of actions. In such environments a compound policy may be learnt with shared neural network parameters, which performs multiple tasks concurrently. However such compound policy may get biased towards a task or the gradients from different tasks negate each other, making the learning unstable and sometimes less data efficient. In this paper, we propose a new approach for simultaneous training of multiple tasks sharing a set of common actions in continuous action spaces, which we call as DiGrad (Differential Policy Gradient). 
The proposed framework is based on differential policy gradients and can accommodate multi-task learning in a single actor-critic network.


We also propose a simple heuristic in the differential policy gradient update to further improve the learning. The proposed architecture was tested on 8 link planar manipulator and 27 degrees of freedom(DoF) Humanoid for learning multi-goal reachability tasks for 3 and 2 end effectors respectively. We show that our approach supports efficient multi-task learning in complex robotic systems, outperforming related methods in continuous action spaces. 




§ INTRODUCTION

There has been an increasing demand for reinforcement learning (RL)<cit.> in the fields of robotics and intelligent systems. Reinforcement learning deals with learning actions in a given environment to achieve a goal. Classic reinforcement learning techniques make use of linear approximation or tabular methods to learn such correlation<cit.>.

With the advancements of deep neural networks in the recent times, learning non-linear approximations and feature extraction has becomes much simpler. It was believed that non-linear approximators like neural network are hard to train in reinforcement learning scenario. However recent advancements in RL has successfully combined the deep neural networks with RL and stabilized the learning process. Deep Q Networks(DQN) <cit.> used Convolutional neural networks(CNN) and fully connected layers to make the RL agents learn to play the ATARI games. Following the success of DQN, several improvements on this architecture like Double DQN<cit.>, Prioritized Replay<cit.>, Duelling Network<cit.> are proposed which propelled the use of Deep RL in multi-agents. <cit.> proposed Deep Deterministic Policy Gradient(DDPG) for continuous control tasks which further extended the scope of Deep RL applications to robotics.


Robotic systems like open and closed kinematic chains offer fresh perspectives to employ deep RL algorithms. <cit.> applied DDPG framework to learn manipulation tasks in complex 3D environments. Following this, <cit.> applied DDPG to learn reachability tasks in Humanoid robot. Even though they are not multi-agent systems, they can be posed as multi-tasking systems where there are shared actions (common kinematic chains). As shown in Fig. 1, the spine/torso is the common chain which contributes to the reachability tasks of both the hands in humanoid robot. In this paper, we propose a novel framework called DiGrad based on differential policy gradients to learn multi-tasking in such robotic systems where different tasks may share a set of common actions. 

There are several mathematical approaches which try to address the problem of multi-tasking in branched manipulators. However, classical methods based on Jacobian<cit.> has very limited capability in branched manipulators. Methods like Augmented Jacobian<cit.> have constrained solution spaces, while methods based on optimization<cit.> often doesn't provide real time control. Hence, RL based solvers are of great use in this domain which can sample the entire solution space and learn a real time controller in such complex robotic systems.

One direction for learning multiple tasks in such scenarios is to use DDPG to learn a compound policy taking care of all the tasks. However we found DDPG to be unstable for such multi-task scenarios. DiGrad addresses this problems by using differential policy gradient updates. We test our framework on branched manipulators shown in Fig. <ref> for learning reachability tasks of multiple end effectors simultaneously. The proposed framework shows substantial improvement over DDPG and is considerably robust on all the experiments we conducted.








The rest of the paper is organised as follows. Section 2 and 3 discusses the related works and background. Section 4 explains the mathematical background behind the proposed framework and provides the detailed algorithm. Finally, Section 5 and 6 contain the experimental results and discussion respectively.



§ RELATED WORKS

Most of the multi-task reinforcement learning algorithms rely on transfer learning approaches. <cit.> shows a good collection of these methods. Some of the recent works based on this approach are <cit.>. Some works by <cit.> and <cit.> explored learning universal abstractions of state-action pairs or feature successors. 

Apart from transfer learning, some works like <cit.>, <cit.> investigated joint training of multiple value functions or policies. In a deep neural network setting, <cit.> provided a framework for simultaneous training of multiple stochastic policies and a distilled master policy. Unlike our work, <cit.> uses multiple networks for each policy and one more network for the distilled policy. In our work, we show how we can use a single network to learn multiple deterministic policies simultaneously.


All the above mentioned methods assume multi-agent scenario whereas in our paper, we concentrate on learning multiple tasks in a robotic system. Some very recent works in this scenario are  <cit.> and <cit.>. These works do not talk about the actions which are shared among different tasks, thus limiting their applicability. Unlike these frameworks, we explore the case of multi-task learning in branched manipulator which have shared action-spaces.


§ BACKGROUND

We consider a standard reinforcement learning setup consisting of an agent interacting with an environment E in discrete time steps. At each time step t, the agent takes a state s_t ∈ S as the input, performs an action a_t ∈ A according to a policy, μ: S → A and receives a reward r_t ∈ R. We assume a fully observable environment and model it as a Markov decision process with state space S, action space A = IR^N, an initial state distribution p(s_1), state transition dynamics p(s_t+1|s_t,a_t) and a reward function r(s_t,a_t).

The goal in reinforcement learning is to learn a policy μ which maximizes the expected return R_t = ∑_t>0γ^tr(s_t,a_t), where γ∈ [0,1] is the discount factor. Since the return depends on the action taken, the policy may be stochastic but in our work we consider only deterministic policies. Hence the discounted state visitation distribution for a policy μ is denoted as ρ^μ.

The action-value function used in many reinforcement learning algorithms is described as the expected return after taking an action a_t in state s_t and thereafter following the given policy:

    Q^μ(s_t,a_t) = 𝔼[R_t|s_t,a_t]

DDPG is an off policy learning algorithm that uses an actor-critic based framework for continuous control tasks.

In DDPG, both actor and critic are approximated using neural networks (θ^μ, θ^Q). The problem of instability in training is addressed by using target networks (θ^μ', θ^Q') and experience replay using a replay buffer. In this setting, the critic network weights are optimized by minimizing the following loss:

    L(θ^Q) = (Q(s_t,a_t|θ^Q) - y_t)^2

where,

    y_t = r(s_t, a_t) + γ Q'(s_t+1,μ'_t+1(s_t+1|θ^μ')|θ^Q')

The gradient ascent update on policy network (actor) is given using Deterministic Policy Gradient(DPG) theorem(<cit.>). Suppose the learning rate is η, then:

    θ^μ = θ^μ + η∇_aQ(s_t,a_t|θ^Q)|_a=μ(s_t|θ^μ)∇_θ^μμ(s|θ^μ)

Finally the updates on target networks are: 

    θ^Q'←τθ^Q + (1-τ)θ^Q'
    θ^μ'←τθ^μ + (1-τ)θ^μ'

where, τ << 1.












In the proposed framework, we use some of the basic concepts of DDPG and use DPG theorem to derive a policy gradient update which can support robust multi-task learning. Finally we explain how the learning can be improved by using a simple heuristic in case of shared actions.



§ DIGRAD: DIFFERENTIAL POLICY GRADIENTS

We propose a framework for simultaneous reinforcement learning of multiple tasks shared actions in continuous domain. The method is based on differential action-value updates in actor-critic based framework using DPG theorem.
The framework learns a compound policy which optimally performs all the shared tasks simultaneously.
Fig. <ref> shows the higher level description of the method.
In this section, we describe the mathematical framework behind this work.



 §.§ Environment setting

Consider n tasks in a standard RL setting and their corresponding n action spaces A_1, A_2, ..., A_n. We will assume that the state space S is the same across all the tasks and an infinite horizon with discount factor γ. Let A be a compound action space which performs the given set of n tasks simultaneously. Let a ∈ A denote compound actions, a_i ∈ A_i denote actions and s ∈ S denote states. Therefore the relation between the compound actions and all the n actions is given by,

    a = ⋃_i=1^na_i.

Suppose a_s corresponds to the set of actions which affects k of the n tasks, then a_s is given by

    a_s = ⋂_j=1^k a_j     where,   a_j ⊂ a


The reward functions for the i^th task depend only on the corresponding actions a_i. Therefore, we denote the reward function as r_i(s,a_i) for each task i; let Q_i(s, a_i) be the corresponding action value function. Let μ be the compound deterministic policy and μ_i be the task-specific deterministic policies; therefore μ_i ⊂μ, where 

    μ_i(s) = a_i   and  μ(s) = a




 §.§ Proposed framework
 
An actor-critic based framework is proposed for multi-task learning in the given environment setting. We use a compound policy μ(s) parametrized by θ^μ, instead of multiple policy networks for each policy μ_i. 

    μ:S → A

A simple parametrization for action-value functions would be to have a separate network θ^Q_i for each Q_i(s,a_i), which outputs an action-value corresponding to the i^th task.
Another approach for modelling action-value function is to have a single network parametrized by θ^Q which outputs action-values for all the tasks. Here, Q_i is the action value corresponding to the i^th task. 
<cit.> showed that a single critic network is sufficient in multi-policy learning. In this setting, the penultimate layer weights for each Q_i are updated based on only the reward r_i(s,a_i) obtained by performing the corresponding action a_i. The remaining shared network captures the correlation between the different actions a_i. Hence this kind of parametrization is more useful in the case of shared actions. Apart from this, the number of parameters are significantly reduced by the use of a single network. 










 §.§ Critic update

Consider a single actor-critic based framework where critic Q(s,a) is given by function approximators parametrized by θ^Q and the actor μ(s) is parametrized by θ^μ. Let the corresponding target networks be parametrized by θ^μ', θ^Q'. Since we have multiple critic outputs Q_i, we optimize θ^Q by minimizing the loss as given by <cit.>:

    L(θ^Q) = ∑_i=1^n (Q_i(s_t,a_it|θ^Q) - y_it)^2

where the target y_it is

    y_it = r_i(s_t, a_t) + γ Q'(s_t+1,μ'_t+1(s_t+1|θ^μ')|θ^Q')




If there are multiple critic networks, network parameters are optimized by minimizing the corresponding loss:

    L(θ^Q_i) = (Q_i(s_t,a_it|θ^Q) - y_it)^2

In both the settings of critic, there is only a single actor which learns the compound policy μ. The differential policy gradient update on the compound policy is explained in the next subsection.  



 §.§ Differential Policy Gradient

Each task has a corresponding reward, r_i(s,μ_i(s)) and hence to learn all the tasks we need to maximize the expected reward for each of the task with respect to the corresponding action, a_i. Therefore the performance objective to be maximized is: 

    J(μ,{μ_i}_i=1^n) = ∑_i=1^n𝔼_s∼ρ^β[r_i(s,μ_i(s))]

The update on the parametrized compound policy μ(s|θ^μ) is given by applying deterministic policy gradient theorem on Eq. (<ref>). The resulting update is:

    ∇_θ^μ J ≈∑_i=1^n𝔼_s∼ρ^β[ ∇_θ^μQ_i(s,a_i|θ^Q)|_a_i=μ_i(s|θ^μ)]


    = ∑_i=1^n𝔼_s∼ρ^β[ ∇_a_iQ_i(s,a_i|θ^Q)|_a_i=μ_i(s|θ^μ)∇_θ^μμ_i(s|θ^μ)]



  §.§.§ DiGrad with shared tasks

In the above environment setting, let all the n tasks share a common set of actions a_s, i.e., k=n. Let a_1, a_2, ..., a_k be the actions corresponding to these k-tasks. Therefore from Eq. (<ref>), 

    a_s = ⋂_j=1^ka_j

Now the compound action a can be written as:

    a = {a_1 ∪ a_2 ...∪ a_k}


    = {{a_1 ∖ a_s}∪{ a_2 ∖ a_s}...∪{ a_k ∖ a_s}∪ a_s }


    = { a_1^d ∪ a_2^d...∪ a_k^d∪ a_s}

where,  a_j^d = { a_j ∖ a_s }


Here we can see that a_1^d, a_2^d, a_k^d, a_s are disjoint sets. Therefore we can write a_i = [a_i^d,a_s]. Similarly, μ_i = [μ_i^d,μ_s]. From here onwards, to make it succinct we drop s∼ρ^β from the subscript of 𝔼_s∼ρ^β and simply represent it as  𝔼.
 

Substituting these in Eq. (<ref>):

    ∇_θ^μ J ≈∑_i=1^k𝔼[ ∇_[a_i^d, a_s]Q_i(s,a_i|θ^Q)|_a_i^d=μ_i^d(s|θ^μ), a_s=μ_s(s|θ^μ)
    ∇_θ^μ[μ_i^d(s|θ^μ), μ_s(s|θ^μ)]]

On expanding with respect to gradient operator we get,

    = ∑_i=1^k𝔼[ ∇_a_i^dQ_i(s,a_i|θ^Q)|_a_i^d=μ_i^d(s|θ^μ);       ∇_a_sQ_i(s,a_i|θ^Q)|_a_s=μ_s(s|θ^μ) ]^T
    [ ∇_θ^μμ_i^d(s|θ^μ);   ∇_θ^μμ_s(s|θ^μ) ]


    = ∑_i=1^k𝔼
    [∇_a_i^dQ_i(s,a_i|θ^Q)|_a_i^d=μ_i^d(s|θ^μ)∇_θ^μμ_i^d(s|θ^μ) 
     + ∇_a_sQ_i(s,a_i|θ^Q)|_a_s=μ_s(s|θ^μ)∇_θ^μμ_s(s|θ^μ)]


    = ∑_i=1^k𝔼
    [∇_a_i^dQ_i(s,a_i|θ^Q)|_a_i^d=μ_i^d(s|θ^μ)∇_θ^μμ_i^d(s|θ^μ)] 
     + 
    𝔼[∑_i=1^k∇_a_sQ_i(s,a_i|θ^Q)|_a_s=μ_s(s|θ^μ)∇_θ^μμ_s(s|θ^μ)]

We can see that the second term on R.H.S of Eq.(<ref>) will be zero if all the action spaces are disjoint, that is, a_s = ∅. Hence this framework can be used even when there are no shared actions. Since the update on the actor are the sum of gradients of different action values, we call this a differential gradient update. It is different from the standard gradient update where an actor is updated based on a single action value <cit.>.


  §.§.§ Heuristic of direction


From Eq.(<ref>), we can see that the policy gradient update for the policy(μ_s) of shared action a_s is taken as sum of the gradients of action values corresponding to the tasks it affects, whereas for policy(μ_j^d) of other actions a^d_j, the gradient update is taken as the gradient of only the corresponding Q_j. Thus, this uneven scaling of gradients may lead to delayed convergence and sometimes biasing. In order to equally scale all the gradient updates, we take the average value of the gradients obtained from the different Q-values for the shared action a_s. This modification will not affect the direction of gradient, only the magnitude will be scaled. By applying this heuristic, the differential gradient update on shared actions becomes:

    ∇_θ^μ J ≈∑_i=1^k𝔼
    [∇_a_i^dQ_i(s,a_i|θ^Q)|_a_i^d=μ_i^d(s|θ^μ)∇_θ^μμ_i^d(s|θ^μ)] 
     + 
    1/k𝔼[∑_i=1^k∇_a_sQ_i(s,a_i|θ^Q)|_a_s=μ_s(s|θ^μ)∇_θ^μμ_s(s|θ^μ)]



  §.§.§ Generalisation

Suppose there are k tasks which share a set of action a_s as above and n-k tasks which are independent, with corresponding actions a_k+1,... ,a_n, then Eq. (<ref>) can be written as:

    ∇_θ^μ J ≈∑_i=1^k𝔼
    [∇_a_i^dQ_i(s,a_i|θ^Q)|_a_i^d=μ_i^d(s|θ^μ)∇_θ^μμ_i^d(s|θ^μ)] 
     + 
    1/k𝔼[∑_i=1^k∇_a_sQ_i(s,a_i|θ^Q)|_a_s=μ_s(s|θ^μ)∇_θ^μμ_s(s|θ^μ)]
     + ∑_i=k+1^n𝔼[ ∇_a_iQ_i(s,a_i|θ^Q)|_a_i=μ_i(s|θ^μ)∇_θ^μμ_i(s|θ^μ)]

From Eq. (<ref>), we can observe that the policy gradient update for the policy (μ_s) of the shared action set a_s is the average of the gradients of the action-value functions of all the tasks it affects. 

This can be easily extended to cases where there are more than one set of shared tasks. Our framework can accommodate heterogeneous dependent action spaces as compared to related multi-task RL algorithms which assume that action spaces are homogeneous or independent or both. This demonstrates the wider applicability of our framework.


 §.§ Algorithm


In this section we explain the algorithm to learn multiple tasks using DiGrad. The flow of the algorithm is very similar to standard DDPG but there are significant differences in terms of the critic and actor updates as shown in the previous subsections. In DiGrad, compound action a is executed in the environment which returns a vector of rewards r⃗_t corresponding to each task instead of a single reward. The replay buffer B stores the current state s_t, compound action a_t, observed state after executing action s_t+1 and the vector of rewards r⃗_t. The entire flow of the algorithm is shown in Algorithm <ref>. 



§ EXPERIMENTS AND RESULTS

The proposed framework was tested in different settings in order to analyse the advantages of each setting. We considered four different network settings for DiGrad as follows:
 (1) Single critic network with heuristics
 (2) Single critic network without heuristics
 (3) Multi critic network with heuristics
 (4) Multi critic network without heuristics. 

We compare all of them with a standard DDPG setting. We use the same set of hyper parameters  in all the five settings. The critic network architecture is the same for both single and multiple critic case in all aspects except in the number of outputs. The actor network parameters are also same for all the cases. We show the comparison of average reward as well the mean scores of each task in all the plots.  Note that the average reward curves for DDPG are not shown as the reward function settings for DDPG is different than that of DiGrad.














In order to test the proposed multi-task learning framework, we considered two different environments. In both these environments, training involved learning reachability tasks for all the end effectors simultaneously, i.e., learning a policy on the joint space trajectories to reach any point in their workspace.
For all the experiments, we define error and score for a particular task i as,



    error_i = ||G_i - E_i||,    score_i = -log(error_i)

where G_i and E_i are the coordinates of goal and end-effector of the i_th chain.

In all the experiments, agents were implemented using TensorFlow Code base consisting of 2 fully connected layers. RMSProp optimizer is used to train both actor and critic networks with learning rates 0.001 and 0.0001 respectively. We used CReLu activation in all the hidden layers. While training, a normally distributed decaying noise function was used for exploration. Whereas, while testing this noise is omitted. We set the discount factor to be 0.99 in all the settings.  In all the tasks, we use low-dimensional state description which include joint angles, joint positions and goal positions.
The actor output is a set of angular velocities q̇. Hence the policy learns a mapping from the configuration space to joint velocity space.


  §.§.§ Reward function


The reward function for DiGrad settings is modelled keeping in mind the multi-task application. As defined before, r_i is the reward corresponding to the action a_i of the i_th task. We give a small positive reward to r_i if task i is finished. Also, if all the end effectors reach their respective goals, a positive reward is given to all the tasks. For all other cases, a negative reward proportional to the error is given. In DDPG setting, there is a single reward unlike DiGrad. A positive reward is given when all the end effectors reach their goals simultaneously. Else, a negative reward is given proportional to the sum of error of all the tasks, that is, sum of distances between the respective goal and its corresponding end effector.
 



  §.§.§ Environments


8-link manipulator: The first environment is an 8 DoF planar manipulator with 3 end effectors as shown in Fig. <ref>. We can observe that the shared sub-chain of 2 joints is common to  all the 3 tasks. Also, the action dimension of the non-shared chains are kept different in order to check the robustness of the framework. The dimensions of each action is given as: a_1 = 3, a_2 = 4, a_3 = 5 and a_s = 2. 

Fig.<ref>(A) shows the performance curves for the five different settings. From the mean score curves, we can see that all the DiGrad based settings converge faster and achieve better final performance than DDPG. Even though the action dimension of each task was different, all the network settings in DiGrad framework worked equally well for all the tasks. Whereas, DDPG showed comparable performance for only Task-2. 


From <ref>(A1), we can see that the single critic framework consistently has better average reward per episode than multi critic frameworks. Thus, modelling all the action value functions in one critic network doesn't affect the performance. In fact, the shared parameters in single critic framework could help the network capture the correlation among the actions much better than multi-critic case. Note that, single critic framework is achieving these performances with lesser parameters than the multi-critic framework. 


DiGrad frameworks with heuristics perform at par with the non-heuristic frameworks. On applying the aforementioned heuristic, no significant improvement in the average reward curve is observed. But in the mean score curves, specially Task-1 and Task-2 curve, we can see that the application of heuristics helps the network to be more stable as compared to their respective non-heuristic curves. Thus, we can say that normalising the gradient of action values of the shared action as in Eq. (<ref>) could help the network deliver robust multi-task training.




Humanoid robot: Secondly, we test our framework on a 27 DoF humanoid robot (Fig. <ref>). This experiment involved reachability tasks of the 2 hands of the humanoid robot using the upper body (13 DoF) consisting of an articulated torso. The articulated torso is the shared chain which is affecting both the tasks. It is noteworthy that the articulated torso has 5 DoF whereas, the arms have 4 DoF each. Thus, the contribution of shared action (articulate torso) to the task is more than the non shared actions (arms). The environment for training is developed in MATLAB and the trained models were tested in a dynamic simulation environment MuJoCo. 

Fig.<ref>B summarizes the results for this case. We found that DPPG is generally unstable for solving multi-tasks problems. In some runs, it may learn initially but suffers degradation later.  We observe that the DiGrad algorithm yields better final results while having greater stability.
 
From the mean scores of the tasks, we can see that the single critic frameworks converge faster and are stable throughout the experiment as compared to the multi-critic frameworks.  The best setting is again the single critic with heuristic, outperforming all the others in all the cases.
 

Note that, the reward function for DDPG is kept different from the DiGrad framework. We also tried a different reward setting taking the sum of individual rewards r_i as a reward signal to DDPG framework, where r_i is same as defined in the DiGrad reward setting.
We observed that this reward setting led to biasing, where one of the tasks dominated others. This behaviour could have been due to the negative interference across tasks, which didn't happen in DiGrad.


§ CONCLUSION

In this paper, we propose a deep reinforcement learning algorithm called DiGrad for multi-task learning in a single agent. This framework is based on DDPG algorithm and is derived from DPG theorem. In this framework, we have a dedicated action value for each task, whose update depends only on the action and reward corresponding to the task. We introduced a differential policy gradient update for the compound policy.

We tested the proposed framework for learning reachability tasks on two environments, namely 8-link manipulator and humanoid. These experiments show that our framework gave better performance than DDPG, in terms of stability, robustness and accuracy.  These performances are achieved keeping the number of parameters comparable to DDPG framework. Also, the algorithm learns multiple tasks without any decrease in the performance of each task. 

Our work focuses on learning coordinated multi-actions in the field of robotics, where a single agent performs multiple tasks simultaneously. The framework was able to capture the relation among tasks where some tasks had overlapped action space. Our future work will focus on extending the framework to multi-agent domain.  
named


