Reward Adaptation Via Q-Manipulation
Transfer RL;Reusable RL;Action Pruning;Reward Adaptation
Thank you for your valuable feedback. Please see our responses to your questions below:

### Clarification of Problem Statement
- The agent doesn’t have access to the underlying MDP when learning the target policy but can interact with the target domain to collect samples. Q-M doesn’t have access to the source data. Source Q-functions (including Q variants for initializing the bounds) and that the target reward function is a known function of the source reward functions is the only information available.

### Motivation for $R = f(R_i)$
- In practice, we often have a rough idea about how the target reward is related to the source rewards. For example, for adapting an autonomous driving agent to drive fast and comfortable when it already knows how to operate either fast or comfortable, we may consider a weighted linear relationship depending on the relative priority of each (a common approach in multi-objective optimization). Such a functional relationship, when unknown,  may also be learned first through regression. 

### Domain Description
- Experiments are designed such that source behaviors are different from target behaviors. For example, in race track individual behaviors correspond to avoid obstacles, terminate, or stay put. The target reward is -0.1 for living reward, -0.2 for colliding into the wall, -1 for staying at the initial location, and +2 for reaching the goal. The goal of the agent in the target domain is to avoid collision and reach the goal faster. Despite the source behaviors being very different from the target, Q-M is able to leverage it and achieve sample efficiency.

### Plots
- When applying RL in practice, we often have sufficient computational capability but access to real-world samples is limited. In the introduction, we stated that “In general, Q-M requires additional computing resources (i.e., CPU time and storage) to implement but its benefits outweigh the costs in practical applications, especially in situations where accessing the target domain for samples is expensive.” Improvement in sample efficiency can be observed in the plots we present in section 3. At the same time, we reported running time comparison in the appendix, which shows that the increase in computation time is manageable. 
    - Time and samples used for policy evaluation in SFQL have not been considered in our results for SFQL; we assume that policy evaluation is essentially free for SFQL (which is true with some bookkeeping from the source behaviors). This also makes it fair to compare SFQL and Q-M since Q-M also assumes bookkeeping from the source behaviors (e.g., Q functions). 

### MORL vs. Q-M
- You are right in the sense that MORL and Q-M share some similarity. When there exists a function that combines the multiple objectives in MORL, the problem setting becomes similar to ours. When that is not possible, MORL introduces a pareto set of policies. Q-M is focused on the transfer learning aspect where source behaviors when available are leveraged to benefit learning the target behavior. MORL, on the other hand, addresses a pure optimization problem.
Thank you for your valuable feedback. Please see our responses to your questions below:

### Quality of the Q Bounds
- It would be interesting to analyze the theoretical relation between the quality of bounds and sample efficiency in the future. In practice, we observed that the quality of bounds affected pruning performance, and hence sample efficiency. As evidence provided in our evaluation (section 3, pages 7-8), we showed that a noisy reward function, which resulted in looser bounds led to pruning fewer actions. However, whether looser bounds always lead to a reduced pruning performance requires a more thorough investigation. 

### Continuous Domains
- Extending Q-M to handle continuous domains (including studying other discretization methods) will be a future challenge so we provided only a simple discretization method to study its impact on the performance of Q-M. As we stated in the paper, discretization has a similar effect as adding noise (potentially unbounded) and hence the performance was expected to be substantially impacted. This observation is motivating us to seek more efficient methods for continuous domains. Currently, we are investigating how to apply Q-M with function approximation and working on a few directions. 

### SFQL vs. Q-M with Source Behaviors Similar to the Target
- For SFQL, if one of the source behaviors is similar to the optimal target behavior,  SFQL would be directly benefiting from it by starting from an almost optimal Q function. However, Q-M uses a very different mechanism for transferring knowledge: it only uses bounds of the Q functions to prune. The similarity between the source and target behaviors does not necessarily translate to tighter bounds. In fact, it could be quite the contrary (e.g., consider the linear case where the upper bound is a summation of all the source Q functions, Eq (9)). It is this very distinct mechanism that allows Q-M to handle cases efficiently (where the source and target behaviors are very different) while SFQL struggled. From this aspect, SFQL and Q-M are complementary to each other and can in fact be applied at the same time to combine their benefits for learning.
Thank you for your valuable feedback. Below, we address the concerns raised in the weaknesses and provide answers to the remaining questions:

### Motivation
- Motivation is a new method of transfer learning for reward adaptation. A real-world example has been provided in the introduction: “adapting an autonomous driving agent to drive fast and safe when it already knows how to operate either fast or safe”. Our approach aims to reduce the samples required for learning the target behavior. It is achieved by pruning out actions that do not contribute to the optimal policy before learning.  Hence, Q-M can be used in an RL system that resembles modular RL where a repository of source behaviors (with additional bookkeeping for Q variants) are maintained and can be reused for learning new behaviors.  
### Q Worst-Case
- It is only needed for initialization. Our iteration method is valid if the bounds are initialized correctly. Other ways that do not require Q worse-case may exist and require future investigation. Eg.  to compute the bounds via the geometric sequence of the highest and lowest rewards (although it is expected to reduce the transferring performance). Assuming Q worse-case for the source behaviors is not necessarily strong assumption. In practice, Q worse-case can be obtained while learning the source behavior (with the same samples used for learning Q*) by leveraging Lemma 1. Storing it requires extra space but is feasible with modern computing. Note that we assume that these worst-case Qs are provided along with the source behaviors (so we do not need to learn them when learning the target behavior). This is not unlike modular RL where a repository of source behaviors can be utilized to form new behaviors but requires additional bookkeeping for the source behaviors (such as their initialization conditions) to apply. 
### Running Time
- Cost depends on the metric used. While you are right that our method is computationally expensive, it is less costly in terms of sample requirement.  Practically in RL, we often have sufficient computational capability but access to real-world samples is limited. In the introduction, we stated that “In general, Q-M requires additional computing resources (i.e., CPU time and storage) to implement but its benefits outweigh the costs in practical applications, especially in situations where accessing the target domain for samples is expensive.” Improvement in sample efficiency can be observed in section 3. 
    - We did not “hide” the computation time results. Line 249 in section 3 highlighted that we reported running time comparison in the appendix, which shows that the increase in computation time is manageable so we did not separately report in the paper (more details below).
### Linear Combination of Rewards
- Linear Combination of Rewards is not a necessary condition for using Q-M. As long as the bounds are initialized correctly, the Q-M iteration remains sound. Linear combination is just a combination function for which we have provided an initialization method. Other efficient initialization methods for other forms of the combination function will be investigated in our future work. 
### Corollary 1
- Corollary 1 is a direct result from Theorem 1 (convergence to a fixed point) and Theorem 2 (non-strict contraction) so we did not provide a proof. Note that the fixed point may or may not be unique.  We have observed different fixed points in evaluation with different initializations as well.
### Training Time Breakdown
- Time taken for target learning via QL in Q-M is similar to that taken by the baseline QL since they are run for the same number of steps to compare. 
- Time taken by the iteration process varies across domains (depending on no. of states, actions, SBF, initialization of bounds, etc). The time taken (in sec) by the iteration process was included in the computation time of Q-M in the tables in the appendix, which is also separately reported below for your convenience. Since the increase is not substantial, we did not separately report it in the paper. 
| Domain                        | SBF_min | SBF_mid | SBF_max |
|-------------------------------|---------|---------|---------|
| Dollar Euro                   | 0.05    | 0.04    | 0.03    |
| Race Track                    | 0.13    | 0.12    | 0.15    |
| Frozen Lake                   | 0.04    | 0.04    | 0.04    |
| Autogenerated                 | 0.11    | 0.20    | 0.18    |
| Autogenerated (randomized MDP and R) | 2.54 | 3.17 | 2.92 |
| Non-linear Target Reward      | 0.95    | 3.83    | 5.25    |
| Noisy Reward Combination      | 2.92    | 2.91    | 2.85    |
### Title
- We perform our computation based on the source Q functions that are assumed to be given to benefit learning the target behavior.  It is a form of knowledge manipulating (instead of directly using) to enable transfer, in our case the Q functions, and hence named _Q manipulation_. 
### Code
- https://tinyurl.com/56pc3avf
We would like to express our sincere gratitude to all the reviewers for their invaluable feedback on our work. 

We would like to point out that, while there are limitations (due to some of the assumptions made), our approach to reward adaptation supports more general knowledge transfer and is fundamentally different from and complementary to the previous work. It can be used by itself to benefit learning the target behavior or as a key part of a larger system that resembles modular RL, and thus has many real-world applications (more details in individual responses). Our main contributions hence lies in theoretically and empirically establishing the feasibility and effectiveness of Q-M and opens up many future research opportunities. We would like to stress that the assumptions only incur manageable additional costs in practice (while allowing Q-M to substantially reduce the samples required for training the target behavior) and are not very different from similar assumptions made in relevant prior works (more details in individual responses). 

Next, we will address individual questions and concerns in our responses to individual reviewers.
Yes
No ethics review needed.
Questions: 
- What are examples of real-world tasks on which your method could be practically useful? (if not in its current form, then at least aspirationally through future extensions)
- What is the break down of the training time using Q-M between the bound refinement and then the subsequent Q-learning?
- Do you disagree with any of the above outlined weaknesses and if so how?
- Why is the method called Q-manipulation? This doesn't seem particularly descriptive of what the method is doing (bounding the Q-function and pruning the actions) - in that light at least half of methods in RL (-adjacent) literature could be called Q-manipulation.


Minor comments and suggestions:
- The notion of "reachable states" is not properly defined/explained upon first usage. I'd recommend explaining as "reachable states" sometimes also refers to states that are eventually reachable (possibly over multiple steps).
- Before, or at least immediately after equations 5, 7, you should comment on the fact that we don't know Q*, otherwise it's really confusing.
- Table 5 in the appendix: I assume |A_p| is the total number of actions left summed over all states. Maybe it would be worth showing this as a percentage of the original number of actions?
- the writing is fairly clear with only minor typos
- the method seems to be a valid method for solving the task as defined (at least in the case where target reward is a linear function of the source rewards)
- the paper provides a valid theoretical result showing that the upper/lower bounds converge to a fixed point.
In the context of tabular RL, the paper addresses the problem of "reward adaptation" - we have multiple policies trained on different source tasks (together with the corresponding optimal Q-values and the Q-values of the worst possible policy for each reward) and we want to produce an optimal policy for the a target task whose reward is a known function of the source rewards. 
The paper proposes a new method for this task called "Q-manipulation", which consists of establishing and then improving a lower and upper bound on the optimal Q-value for the target task. Where the lower bound for some actions is higher than the upper bound for other actions, this then allows us to prune the action space in each state, so when subsequently solving for the optimal Q-value on the target task, we have an easier task to solve. 
The paper gives a Bellman iteration algorithm for refining the upper and lower bounds and shows that it converges to a (non-unique) fixed point. They also evaluate on several simple simulated tasks against Q learning without any reward adaptation and one prior method with reward adaptation.
- The importance of task is not addressed so the paper lacks good motivation. No example of a practical real-world task where this could be useful is given.
- This becomes even more pressing given the assumptions of the proposed method. Firstly, even before the method is applied, it assumes we have access to the Q-values of the worst possible policy for each source task (in addition to the optimal ones). However, such worst-case Q-values are not something we usually naturally have in hand, and in general, obtaining each such worst-case Q-function means solving the whole RL problem as many times as there are source tasks. Instead, we could just solve for the target task and be done.
	- one could argue that this would be worth it in cases where we have a fixed set of source tasks, and we need to repeatedly adapt to different combinations of those, but this is not even mentioned.
- Ok, assume that we for some reason have the worst-case Q-functions. Then, the proposed method still involves running a form of value iteration to refine the upper and lower bound on the target optimal Q-values. The paper doesn't show that this is cheaper than solving for the target task directly. Furthermore, for the method to make sense, it needs to be cheaper to first solve for this and then also run the subsequent Q-learning.
- The timing results hidden in the appendix show that on most tasks, this is indeed slower than running Q-learning from scratch, defeating the purpose of the method.
- Furthermore, I see hiding these important negative results in the appendix without mentioning them in the main text dishonest. The main article shows training curves with iterations that, for the proposed method, include only iteration spent after the expensive pre-training, which may give a wrong impression of computational savings.
- A fully valid end-to-end method is provided for target rewards that are linear combinations of the source rewards - a fact that is again somewhat hidden in the text. I would like to see that emphasized earlier as it is an important limitation on the contributions of the paper.
- Corollary 1 is never proven. To prove it, you can just give an example of an MDP with non-unique fixed points, which could also serve as a useful illustration.
- Not enough details are provided for reproducing the experiments. Code is not provided.
Yes
No ethics review needed.
In Section 3.4, the following statement,

"In Pong, SF-DQN outperformed both QM-DQN and DQN. This was due to the choice of
source behaviors that are either keeping left or right. The target behavior requires the agent to move
to the left and right to catch the ball, which shares strong similarity with the source behaviors."

is confusing. Why doesn't the similarity between source and target functions also help the Q-M agent? Shouldn't the effects be felt more with QM-DQN than SF-DQN? If there is some insight here, a discussion on why QM-DQN didn't perform well would be useful to future readers as a gauge for determining which kinds of problems would be solved with either method.
Overall, the strongest feature of this work is the theoretical guarantees provided on the convergence to a fixed point using the iterative method. This shows that the method is mathematically sound and motivates the extensions to noisy combination functions and continuous action/state spaces.

Pruning the unnecessary actions has the effect that the efficiency of the algorithm decreases with the stochastic branching factor of the MDP, leading to an interesting analysis of the SBF's contribution to convergence speed. As expected, convergence is instant with an SBF of 1, which is already an improvement over SFQL.

Empirically, Q-M outpaces the baselines in the discrete setting, with the linear combination function results being the most convincing. The robustness of the method to noisy reward functions is also convincing as the knowledge of the target reward function may be considered too restrictive in any practical settings.

The clarity of the work is good and the communications of the authors' claims and results are well-formulated.
This work proposes Q-Manipulation, a method for reward adaptation in RL which manipulates a Q-function to allow agents to adapt to new reward functions using prior learned behaviours. Upper and lower bounds for Q* for the new task can be determined when the combination function (the transform between the existing reward functions and the new one), which can be iteratively tightened and are guaranteed to reach a fixed point, similar to value iteration.
The robustness of the method depends on the accurate initialisation of the Q bounds, and improperly placed bounds (which may occur if the target reward function is not well understood, for e.g.) might lead to inefficiency in the iterative pruning process. An analysis of the quality of the Q bounds to the final convergence speed in practice would be interesting.

Unfortunately, the good results in discrete environments don't carry over well to the continuous domain. The authors admit that the results for QM-DQN are only 'marginally better' for 2 environments, but even this is debatable for Lunar Lander. This is a crucial question for the future of this work because problem settings with known reward functions and convenient state discretisations may be difficult to find. In addition to the degradation of performance as the SBF increases, the scope of environments where Q-M is a viable solution seems limited. Have the authors considered other discretisation schemes, or perhaps a way to use Q-M in continuous environments without discretisation?

Finally, the lack of baselines makes the method a little hard to evaluate in the wider context of reward adaptation. However, as I was unable to identify any appropriate baselines apart from those in the paper, I cannot hold this against the authors as RA is largely an unexplored field as of yet.
Yes
No ethics review needed.
When the target reward is a linear combination of the source rewards, this setup appears quite similar to multi-objective reinforcement learning (MORL), where each objective is summed linearly. Could you clarify if there is any relationship between your method and multi-objective RL?
+ Theoretical analysis on the Q bound
This paper considers the reward adaptation in reinforcement learning. The task is to learn optimal policy for target reward function given behavior data under source reward. The method is developed based on the assumption that the target reward is a known function of the source reward to derive the upper and lower bound of the Q function to perform action pruning. The proposed method is compared with successor feature based reward adaptation method in several domains.
+ Problem Definition: The problem the authors aim to address is not clearly described. In Definition 1, reward adaptation is defined as the task of learning an optimal policy for a target reward function, given a set of behaviors trained under source reward functions. However, it is unclear whether the agent still has access to the underlying MDP when learning the target policy. Is the proposed method an offline algorithm where the target policy is learned only based on the source data?

+ Assumption. The assumption that the target reward function is a known function of the source rewards (e.g., a linear combination) is not well-motivated and seems overly restrictive. Additionally, the experimental domains are not well-described, making it difficult for readers to interpret this assumption. For instance, in Race Track, $R_3$ assigns a positive reward (+3) for remaining at the initial location.. This is a relatively large reward compared to others in $R_1$ and $R_2$. In the target domain when $\mathcal{R}=R_1+R_2+R_3$, factors like the distance between the goal and initial location, maximum episode length, and discount factor significantly influence policy behavior. In extreme cases, such as with a short maximum episode length and low discount factor, the optimal policy could be to remain at the initial location.

+ Plots. For the Q-M method, pre-training and pre-computation of Q-functions under the source reward are required. However, in the visualizations, it appears that the computational costs associated with this stage are not included in the plots. This omission raises concerns about the fairness of the comparisons.
Transfer RL approach for Reward Adaptation via action pruning
@inproceedings{
anonymous2024reward,
title={Reward Adaptation Via Q-Manipulation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ZK4VSRzBNC},
note={under review}
}
In this paper, we propose a new solution to reward adaptation (RA), the problem where the learning agent adapts to a target reward function based on one or multiple existing behaviors learned a priori under the same domain dynamics but different reward functions.  RA has many applications, such as adapting an autonomous driving agent that can already operate either fast or safe to operating both fast and safe. Learning the target behavior from scratch is possible but often inefficient given the available source behaviors. Our work represents a new approach to RA via the manipulation of Q-functions.  Assuming that the target reward function is a known function of the source reward functions, our approach to RA  computes bounds of the Q function. We introduce an iterative process to tighten the bounds, similar to value iteration. This enables action pruning in the target domain before learning even starts. We refer to such a method as Q-Manipulation (Q-M). We formally prove that our pruning strategy does not affect the optimality of the returned policy while empirically show that it improves the sample complexity. Comparison with baselines is performed in a variety of synthetic and simulation domains to demonstrate its effectiveness and generalizability.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Transfer RL Reusable RL Action Pruning Reward Adaptation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c94660e292cedb24887538184977b2268846bce0.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reward Adaptation Via Q-Manipulation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission