Relationship Explainable Multi-objective Reinforcement Learning with Semantic Explainability Generation
Huixin Zhan Yongcan Cao




Unitary vs pseudo-unitary time evolution and statistical effects in the dynamical Sauter-Schwinger process
    J. Z. Kamiński
    December 30, 2023
==========================================================================================================

empty
empty





Solving multi-objective optimization problems is important in various applications where users are interested in
obtaining optimal policies subject to multiple, yet often conflicting objectives. A typical approach to obtain optimal policies is to first construct a loss function that is based on the scalarization of individual objectives, and then find the optimal policy that minimizes the loss. However, optimizing the scalarized (and weighted) loss does not necessarily provide guarantee of high performance on each possibly conflicting objective because it is challenging to assign the right weights without knowing the relationship among these objectives. Moreover, the effectiveness of these gradient descent algorithms is limited by the agent's ability to explain their decisions and actions to human users. The purpose of this study is two-fold. First, we propose a vector value function based multi-objective reinforcement learning (V2f-MORL) approach that seeks to quantify the inter-objective relationship via reinforcement learning (RL) when the impact of one objective on others is unknown a prior. In particular, we construct one actor and multiple critics that can co-learn the policy and inter-objective relationship matrix (IORM), quantifying the impact of objectives on each other, in an iterative way. Second, we provide a semantic representation that can uncover the trade-off of decision policies made by users to reconcile conflicting objectives based on the proposed V2f-MORL approach for the explainability of the generated behaviors subject to given optimization objectives. We demonstrate the effectiveness of the proposed approach via a MuJoCo based robotics case study.







§ INTRODUCTION


In recent years, the application of RL in tasks with high-dimensional sensory inputs has shown the potential of creating artificial agents that can learn to accomplish a number of challenging tasks, including the Atari games <cit.>, self-driving cars <cit.>, and Go <cit.>. However, the approaches developed therein mainly focus on finding a single usable strategy, without considering the trade-off among potential alternatives that can increase one objective's value at the cost of another.

In the multi-objective setting, the completion of a task requires the simultaneous satisfaction of multiple objectives such as balancing the power consumption and performance in Web servers <cit.>. Such problems can be modeled as multi-objective Markov decision processes (MOMDPs) and solved by some existing multi-objective reinforcement learning (MORL) algorithms <cit.>. However, solutions obtained via these approaches can hardly balance the possibly conflicting objectives to achieve satisfactory performance on all objectives.

Recently, several interesting MORL approaches have been developed. The author in <cit.> proposed the use of both linear weighted sum and nonlinear thresholded lexicographic ordering methods to develop a multi-objective deep RL framework that includes both single- and multi-policy strategies. The author in <cit.> proposed an architecture in which separated deep Q-networks (DQNs) are used to control the agent's behavior with respect to particular objectives. Then, each DQN has an additional decision value output that acts as a dynamic weight used while summing up Q-values. The authors in <cit.> used softmax-epsilon selection based on a nonlinear action-selection operator. The agents incorporate an action-selection function that is defined as an ordering over these Q-values. In summary, most of the algorithms are based on the scalarization method to transform the multi-objective problem into a single objective one. The scalarization can be nonlinear or linear <cit.>. Other advanced methods include, e.g., the convex hull <cit.>, the varying parameters approaches <cit.>, the constraint method <cit.>, the sequential method <cit.>, and the max-min method <cit.>.

The authors in <cit.> proposed an upper bound for the multi-objective loss and proved that optimizing this upper bound via gradient-based multi-objective optimization yields a Pareto optimal solution. The Frank-Wolfe solver is used to find a minimum-norm point in the convex hull of the set of input points. This work provides a new perspective for balancing objectives when the values for all objectives are considered as the min-norm points in the convex hull. This work showed success in large scale multi-label learning tasks. However, it is unclear if the method can be extended to complex continuous space planning tasks.

When the values for all possibly conflicting objectives are considered as a vector and balancing them is required, it is critical to train a policy and explain why a particular behavior is generated is the topic under study. To address the critical issue, this paper focuses on proposing an explainable V2f-MORL approach. The proposed research has three main contributions. First, we propose an approximate optimistic linear support algorithm (AOLS), which allows the quantification of inter-objective relationship using the inter-objective relationship matrix (IORM). Second, instead of using scalarized Q-value and the action selection approach based on the priority objective value, the proposed method supports vectorized objective state values. In particular, we propose the creation of multi-objective value functions that can be used sequentially in the training of the critics to update the objective state values and the training of the actor to update the control policy. Third, our method is applicable in high-dimensional continuous action spaces with an explainable planning via natural language representation. To our best knowledge, this is the first time that actor critics with quantifiable inter-objective relationship are developed to solve MORL with semantic representation. We also show via one MuJoCo example that the proposed method outperforms the existing single objective optimization methods.




§ PRELIMINARIES



 §.§ Multi-Objective Value Function

For a control policy π, we here propose the construction of a set of multi-objective value functions 𝒴_i^π, i=1,⋯,I via our defined IORM W as 𝒴^π=[𝒴_1^π, ⋯, 𝒴_I^π]∈ℝ^I = W 𝐕^π, where W∈ℛ^I× I is the IORM, I is the number of objectives, 𝐕^π=[V_1^π,⋯,V_I^π]^T with V_i^π representing the state value for the ith objective subject to the control policy π. The ith row of W, given by [w_i1,⋯,w_iI], characterizes how other objective values, {V_j^π|j≠ i}, impact the ith objective value V_i^π under the policy π. 
We here provide a formal definition of the multi-objective value function under the policy π.


Each multi-objective value function is a cumulative sum of objective state values with additive specific impact elements of the form given by

    𝒴_i^π[i_k]
    =∑_j=1^I w_ij[i_k]V_j^π[i_k],   i=1,⋯,I,

where w_ij is the weight quantifying the impact of V^π_j on V^π_i, i_k is the time step number, and k is the number of sequence.






 §.§ Inter-objective Relationship Matrix (IORM)

Based on Definition <ref>, we define the inter-objective relationship matrix (IORM) as

    W=  [ w_ij ]∈ℝ^I× I.
 

Because the impact of one objective on another objective is unknown a priori, an IORM can be assigned an initial value but it needs to be updated based on the input observation, denoted as 𝒳, and a collection of objective value spaces, denoted as {𝒴^i}_i∈ | I  |, for each objective value y^i with m input/output examples denoted as (x^i_1,y^i_1),....,(x^i_m,y^i_m). In particular, IORM will be updated via numerous batches. During each batch,  I× k time steps will be divided into I sequences. In the i_kth time step, m examples are trained to fit y^i, which is then used to update the objective state values V_i. {V_j^π[i_k]}_j∈ | I  | is then used to update the ith row of the relationship matrix W. More detailed description of such an update process will be provided in Section <ref>.




As a consequence, the vector value function 𝒴^π=[𝒴_i^π]∈ℝ^I× I, where 𝒴_i^π is defined in (<ref>), can be updated via

    [ 𝒴_1^π[k];        ⋮; 𝒴_n^π[k] ]
    =W[k][ V_1^π[k];        ⋮; V_n^π[k] ],

where W[k] is the updated IORM at the kth time step. 




§ PROPOSED METHOD



 §.§ Multi-objective Decision Making

In this paper, we consider the problem when multiple objectives O_i, i=1,⋯,I, need to be optimized for a given mission, where I denotes the number of objectives. For example, in robotic locomotion, maximizing forward velocity but minimizing joint torque and impact with the ground, result in a very large number of options to consider. We use 𝐕^π∈ℝ^d×1 =  [V_1^π,⋯,V_I^π]^T, where d ≥ 2, to represent the vector value function for O_i, i=1,⋯,I, subject to the control policy π. A typical approach to optimize objectives O_i, i=1,⋯,I, is to construct a scalarized value function of the form V_w^π∈ℝ^1 =w𝐕^π, where w=[w_1,⋯, w_I] satisfying w1=1, 1 is an all-one column vector, and weight w_i specifies how much each objective contributes to the scalarized objective. A more general form of the scalarized value function is given by V_w^π=f(w, V_1^π,⋯,V_I^π)  <cit.>, where f(·,·) is a nonlinear function. Hence, a multi-objective optimization problem can be converted to a single-objective optimization problem.

These value functions map a multi-dimensional policy value to a scalar according to the preferred policy on decision making and preference elicitation. Since all objectives are desirable, V_w^π is monotonically increasing in all objectives. Given this monotonicity property, the solution set is a Pareto front, i.e., convex coverage set (CCS), that contains for any allowed policy π^' with value V^', any policy that has a greater or equal value in all objectives <cit.>. CCS can quantify the relationship between different policies subject to a number of objectives because one policy can yield good performance for one objective while poor performance for another objective. For instance, for objective i_1, solution θ is better when the loss ℒ̂^i_1 ( θ_V_i_1, θ_V_j_1 ) = - V_w^π < ℒ̂^i_1 ( θ_V_i_1, θ_V_j_1 ), while for objective i_2, solution θ is better when ℒ̂^i_2 ( θ_V_i_2, θ_V_j_2 )>  ℒ̂^i_2 ( θ_V_i_2, θ_V_j_2 ). CCS defines the set of vector objective state values that the optimal value must reside in because every other vector objective state value not in the set will not be the best choice since there exists at least one vector in CCS that is not smaller than it. 

A formal definition of the convex coverage set is given below.

The convex coverage set, denoted as CCS, is the set of all actions and associated payoff values that are optimal for some w of the scalarization function f(w, V^π):

    ∃ V ∈ CCS: ∀ V^'∈π: w^TV^'≤ w^TV,∀ w ∈ℝ^d,

where u_w(·) is the value when taking action a based on the weight w.




 §.§ Multi-objective Reinforcement Learning (MORL)


We first introduce a few definitions that are needed in solving multi-objective optimization problems using (deep) reinforcement learning. Let a trajectory τ^i={q(x_1),q(x_t+1|x_t,a_t),H} consist of a distribution over initial observations q(x_1) with a transition distribution q(x_t+1|x_t,a_t) and an episode length H. We define the loss ℒ(x_1,a_1,...,x_H,a_H) as the negated expected accumulated reward for a series of state-action pairs with length H given by

    ℒ^i(f_θ_π)=-E_x_t,a_t∼ f_θ_π,τ^i∑_t=1^HR_i(x_t,a_t),i=1,⋯,I

where E is the expectation operation, f_θ_π,τ_i is the action distribution function determined by the policy π that is assumed to be constructed using a neural network with θ_π acting as the weights. Another set of hyperparameter is needed to quantify the map from V_i^π to 𝒴^i defined as

    f(V_i^π;w_ii,w_ij): V_i^π→𝒴^i,

where w_ii and w_ij are the weights in the ith row of the IORM W.



 §.§ W Update




We now provide a detailed description of how W is updated within a batch. First, let's define the map from 𝒳 to {𝒴^i}_i∈ | I  | in a parametric form as f^i(x;θ_V_i,θ_V_j) : 𝒳→{𝒴^i}_i∈ | I  |, where θ_V_i and θ_V_j are the aggregated weights of weights θ_π and hyperparameters w_ij. The main idea to update W is to first obtain the CCS, then evaluate the marginal weights 

on the CCS, and finally use the best marginal weight to update W.

Because it is difficult to obtain the CCS directly, we employ the approximate optimistic linear support (AOLS) approach <cit.> to get an approximated set. The AOLS is a method that can gradually improve the approximation of the CCS. Given a maximum improvement threshold ε>0, the AOLS algorithm can compute an approximated ε-optimal set, denoted as CCS, which may diverge from the optimal undominated set by at most ε. Consequently, its marginal weight can be obtained. 
Before a complete undominated set is obtained, a partial CCS can be obtained by evaluating the largest improvement for weights via the priority queue of the marginal weight in this step.
An element in the vector value function over a partial CCS is defined by 
V_S^* ( w  )=max_V^π∈ S  w· V(s,ϕ_V),
where S is the partial CCS, V(s,ϕ_V) is the approximated objective state value vector based on the current critic networks using the current weights ϕ_V, and s is the current state.


AOLS always selects the marginal weight w that maximizes an optimistic upper bound on the difference between V_CCS ( w  ) and V_S^* ( w  ), i.e., V_CCS (w  )- V_S^* ( w  ), which can be updated iteratively to obtain a more accurate max_w  V_CCS(w). The pseudocode for AOLS is shown in the Algorithm <ref>. 






 §.§ Value-function and Policy update
 
To obtain policy network and value function approximation network, we propose to adopt an actor-critic network with one actor network and I critic networks, where the actor network is used to maximize the objective state value and each critic network is used to map from the state action pair to 𝒴_i^π. Assume that the actor network with weights θ_π generates actions via
a = π(s; θ_π).
The weights θ_π can be updated using policy gradient given by <cit.>:

    Δθ_π   ∼∑_k▽_θ _πlogπ_θ ( s_k,a_k ) δ_k,t,

where δ_k,t is the expected value of the ith objective, also known as the temporal difference (TD) residual of V^π_i with discount γ <cit.>, given by

    δ_k,t =    r_i ( s_k,t,a_k,t )+γV_i^π(θ_π) ( s_k,t+1;ϕ_V_i )
       -V_i^π( θ^-_π) ( s_k;ϕ^-_V_i )

where r_i ( s_k,t,a_k,t ) is the immediate reward at the tth time step on the kth experience, V_i^π( θ^-_π) ( s_k,t;ϕ^-_V_i ) is the approximation of the value function V_i based on the old weights θ^-_π for the actor network and the old weights ϕ^-_V_i for the ith critic network, and V_i^π(θ_π) ( s_k+1;ϕ_V_i ) is the approximation of the value function V_i based on the updated weights θ_π for the actor network and the updated weights ϕ_V_i for the ith critic network.



For the I critic networks, its ith neural network with hyperparameter ϕ_V_i is used to approximate each element in the vector value function  V_i^π(s). Assume that the critic function is given by V_i(s;ϕ_V_i) with ϕ_V_i serving as the weights. The weights can be updated via
Δ _kϕ _V_i∼ -▿_ϕ _V_i∑_kδ^2_k,t.



In the standard TD-residual method, the value of one action evaluated via (<ref>) is an incremental form of value iteration. The key drawback of the standard TD-residual method includes the need for a large number of samples and large variance of policy gradient estimate. To address these issues, an existing approach, called generalized advantage estimator (GAE) <cit.>, can be used to evaluate the action advantages and perform the policy updates using proximal policy optimization <cit.>. The GAE is defined by:

    A_t^GAE(γ ,λ)
    
    =   lim_H→∞ ( 1-λ ) ∑_j=1^H λ^j-1∑_k=1^j γ^j-1δ_k,t+j-1
    
    =   ∑_l=0^H(γλ)^lδ_k,t+l,

where λ∈[0,1] and γ∈[0,1] adjusts the bias-variance tradeoff of GAE. 
 
After new weights of the advantage actor-critic network models are obtained, V^π can be obtained via new samples using the updated policy. Afterwards, the procedure in Subsection <ref> can be implemented to obtain the updated W. The entire process will iterate until V_CCS ( w  )- V_S^* ( w  )<ϵ, where ϵ is a small threshold selected by users. 



 §.§ Explainable Planning Representation
 
To address the quantifiable inter-objective relationship in our algorithm, we adopt an explainable planning representation that enables automatic explanation of the planning rationale.


  §.§.§ Vocabulary for Quality Attributes (QA)
 We map QA analytic models to domain-specific vocabulary to be used to generate verbal explanation. The vocabulary includes “QA type”, “optimization objective”, and “QA property” for the description of standard QAs.



  §.§.§ QA Language Templates
 To generate verbal explanation of the objectives and the QA properties of a solution policy π^⋆, we use predefined natural-language templates. Table <ref> shows an example of verbal explanation of QA objectives and properties.





  §.§.§ Obtaining Alternative Policies


Algorithm <ref> outlines an approach for sampling alternative policies around the current policy. The key idea of the approach is to start with the QA values of the current solution policy π: V_1^π(s),⋯,V_I^π(s). For each QA i, we determine a new value V_i^' that is more preferable than V_i^π(s). Then, we construct a new planning problem with I-1 optimization objectives (namely, excluding the objective associated with the QA i), resulting in a new multi-objective value function subject to the constraint that the QA i must be at least as good as V_i^'. Next, we select an optimal, constraint-satisfying solution value under π^' for the new planning problem. The new policy (respectively, state value) provides an alternative of the current policy (respectively, state value associated with the current policy). This procedure will be executed iteratively until we obtain up to M_i number of alternative policies for each i. The pseudocode of the algorithm is given below.





  §.§.§ Semantic Explanation of Value Tradeoffs
 Our value justification indicates the amount of gain-loss in the QAs if one were to choose each alternative value under the current policy. It then indicates preference towards the current policy by arguing that such gain is not worth the loss, reflecting the QA utility models underlying the multi-objective value function. We use a predefined natural language template for generating verbal justification: “I could [improve these QAs to these values], by [carrying out this alternative policy] instead. However, this would [worsen these other QAs to these values]. I decided not to do that because [the improvement in these QAs] is not worth [the deterioration in these other QAs]".



 §.§ Overall Algorithm

The pseudocode for the proposed V2f-MORL approach described in Subsections <ref>, <ref>, and <ref> is given in the Algorithm <ref>.



§ EXPERIMENTS



 §.§ Setup

We here select the testing environment Ant-v2 on the MuJoCo physics engine <cit.>, and select four objectives: Reward Control (Rctrl), Reward Contact (Rcont), Reward Survive (Rsurv), and Reward Forward (Rfor). We use the proximal policy optimization clipping algorithm with ϵ=0.2 as the optimizer. The discounting factor is selected as γ=0.99. One episode, characterizing the number of time steps of the vectorized environment per update, is chosen as 2048. For stabilization purposes, we execute parallel episodes in one batch. The batch size is chosen as the product of the episode size and the number of environment copies simulated in parallel. The number of environment copies is selected as 8. The parameters are optimized using the Adam algorithm <cit.> and a learning rate of 3×10^-4. All of the experiments were performed using TensorFlow, which allows for automatic differentiation through the gradient updates <cit.>. 

The parametric form f^i(x;θ_V_i,θ_V_j) : 𝒳→{𝒴^i}_i∈ | I  |, where θ_V_i is the objective specific weight and θ_V_j is the inter-objective weight, is a CNN whose structure is shown in Fig. <ref>. The architecture specification is given in Table <ref>.  The FC8 and FC4 correspond to the 8-action policy π(·|s_t) and the value function 𝐕(s_t). In the experiments, the screen is resized to an 84× 84×3 RGB image as the network input.

  
   





 §.§ Accuracy vs Episodes




We further investigated the effects of the number of training episodes (including series of time steps) on the maximal relative improvement of the CCS. Fig. <ref> shows how the maximal relative improvement Δ_r(w)=V_CCS(w) - V_S^*(w)/V_CCS(w) of the CCS evolves with respect to the number of episodes. It can be seen from Fig. <ref> that the error is highly affected by the number of training episodes. Although the proposed method is unable to provide sufficient accuracy to build the CCS initially, the deviation will gradually decrease to 0 when the number of episodes is 40. 




 §.§ Testing Results and Discussion

For the simplicity of presentation, we call our proposed method PPO_CCS method. To show the benefit of the proposed PPO_CCS method, we show the results when (1) multi-objective optimization is solved via one single-objective optimization, and (2) the marginal weight in our method is replaced by the corner points of the CCS. All these results are based on the MuJoCo simulator <cit.>.


Our goal is to make a four-legged ant-v2 walk forward as fast as possible while saving cost simultaneously. More specifically, our goal is to maximize the reward forward and the reward survive while minimizing the reward control and the reward contact. We take the current reward function in the OpenAI Gym environments as a baseline, use the cumulative reward trained by the single objective PPO as a benchmark <cit.>, and compare it with our proposed method.

Table <ref> shows the rewards using PPO_CCS and PPO with single objective. It can be observed that the proposed PPO_CCS yields higher reward. It can also be observed that PPO_CCS can generate higher rewards in most cases because PPO_CCS can optimize multiple objectives simultaneously, while the PPO with single objective does not seek to optimize multiple objectives. 



   


 §.§ Natural Language Representation Demonstration


We demonstrate three semantic representations that are generated under different policies using the proposed algorithm in Algorithm <ref>. The result is shown in Fig. <ref>. The first representation provides verbal explanation of the state values under the selected policy while the second and third representations provide verbal explanations why alternative policies were not selected. This is because the algorithm: (i) searched to reduce Rctrl on the CCS and found a different multi-objective value function that increases Rcont, decreases Rsurv, and decreases Rfor, and (ii) searched to reduce Rcont and found a different multi-objective value function that increases Rctrl, decreases Rsurv, and decreases Rfor.

   
   





§ CONCLUSIONS


In multi-objective optimization problems, the possibly conflicting objectives necessitates a trade-off when multiple objectives need to optimize simultaneously. A typical approach is to minimize a loss of weighted linear summation of all objective functions. However, this approach can hardly guarantee good performance on individual objectives because it is very difficult to determine the right weights due to the lack of knowledge in inter-objective relationships. To address the challenge, we proposed a vector value function based multi-objective deep reinforcement learning to solve high-dimensional multi-objective decision making problems. The proposed method optimizes vectorized proxy objectives sequentially based on proximal policy optimization, actor-critical network, and the derivation of optimal weights via marginal weight. 


By explicitly quantifying inter-objective relationship via relationship matrix, the relative importance of the objectives unknown a prior can be obtained via reinforcement learning. Each entry in the relationship matrix specifies and explains the relative impact of one objective on another objective in the optimization step. Moreover, in order to address the interpretability of the proposed V2f-MORL approach, we proposed a new approach to generate alternative multi-objective values/policies to automatically explain the rationale behind decided actions/policies.




                                  
                                  
                                  
                                  
                                  













IEEEtran





